2024-05-01 11:42:36,410 [ 214877 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy (runner:42, check_args_and_update_paths)
2024-05-01 11:42:36,410 [ 214877 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:90, check_args_and_update_paths)
2024-05-01 11:42:36,410 [ 214877 ] INFO : src dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/src (runner:97, check_args_and_update_paths)
2024-05-01 11:42:36,410 [ 214877 ] INFO : base_configs_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration  (runner:99, check_args_and_update_paths)
2024-05-01 11:42:36,410 [ 214877 ] INFO : Unknown image altinityinfra/dotnet-client (runner:357, <module>)
Running pytest container as: 'docker run   --rm --name clickhouse_integration_tests_uuiajx --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_BASE_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_KERBERIZED_HADOOP_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_KERBEROS_KDC_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_JS_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_NGINX_DAV_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600  -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_force_recovery/test.py::test_cluster_recovery test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_internal_secure/test.py::test_secure_raft_works test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart test_keeper_reconfig_add/test.py::test_reconfig_add test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 test_keeper_s3_snapshot/test.py::test_s3_upload test_keeper_secure_client/test.py::test_connection test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_library_bridge/test.py::test_load_all test_library_bridge/test.py::test_load_all_many_rows test_library_bridge/test.py::test_load_ids test_library_bridge/test.py::test_load_keys test_library_bridge/test.py::test_null_values test_library_bridge/test.py::test_path_validation test_library_bridge/test.py::test_recover_after_bridge_crash test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_lost_part/test.py::test_lost_last_part test_lost_part/test.py::test_lost_part_mutation test_lost_part/test.py::test_lost_part_other_replica test_lost_part/test.py::test_lost_part_same_replica test_max_http_connections_for_replication/test.py::test_keepalive_timeout test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 'test_merge_tree_s3/test.py::test_alter_table_columns[node]' 'test_merge_tree_s3/test.py::test_attach_detach_partition[node]' 'test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]' 'test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]' 'test_merge_tree_s3/test.py::test_freeze_unfreeze[node]' 'test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]' 'test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]' 'test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]' 'test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]' 'test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]' 'test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]' 'test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node]' 'test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]' 'test_merge_tree_s3/test.py::test_table_manipulations[node]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]' test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users 'test_multiple_disks/test.py::test_alter_double_move_partition[DISK]' 'test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME]' 'test_multiple_disks/test.py::test_alter_move[mt]' 'test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK]' 'test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME]' 'test_multiple_disks/test.py::test_alter_policy[mt]' 'test_multiple_disks/test.py::test_alter_policy[replicated]' 'test_multiple_disks/test.py::test_background_move[mt]'  -vvv" altinityinfra/integration-tests-runner:0-8e9be16153250c8307730c9b88c8517181f494ce '.
Start tests
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /ClickHouse/tests/integration
configfile: pytest.ini
plugins: xdist-3.5.0, repeat-0.9.3, random-0.2, order-1.0.1, timeout-2.2.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
created: 10/10 workers
10 workers [100 items]

scheduling tests via LoadFileScheduling

test_merge_tree_s3/test.py::test_alter_table_columns[node] 
test_lost_part/test.py::test_lost_last_part 
test_keeper_four_word_command/test.py::test_cmd_clrs 
test_keeper_persistent_log/test.py::test_ephemeral_after_restart 
test_multi_access_storage_role_management/test.py::test_quotas 
test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 
test_max_http_connections_for_replication/test.py::test_keepalive_timeout 
test_log_family_s3/test.py::test_log_family_s3[Log] 
test_library_bridge/test.py::test_load_all 
test_multiple_disks/test.py::test_alter_double_move_partition[DISK] 
[gw0] [  1%] PASSED test_keeper_four_word_command/test.py::test_cmd_clrs 
test_keeper_four_word_command/test.py::test_cmd_conf 
[gw0] [  2%] PASSED test_keeper_four_word_command/test.py::test_cmd_conf 
test_keeper_four_word_command/test.py::test_cmd_cons 
[gw0] [  3%] PASSED test_keeper_four_word_command/test.py::test_cmd_cons 
test_keeper_four_word_command/test.py::test_cmd_crst 
[gw0] [  4%] PASSED test_keeper_four_word_command/test.py::test_cmd_crst 
test_keeper_four_word_command/test.py::test_cmd_csnp 
[gw0] [  5%] PASSED test_keeper_four_word_command/test.py::test_cmd_csnp 
test_keeper_four_word_command/test.py::test_cmd_dump 
[gw0] [  6%] PASSED test_keeper_four_word_command/test.py::test_cmd_dump 
test_keeper_four_word_command/test.py::test_cmd_isro 
[gw0] [  7%] PASSED test_keeper_four_word_command/test.py::test_cmd_isro 
test_keeper_four_word_command/test.py::test_cmd_lgif 
[gw0] [  8%] PASSED test_keeper_four_word_command/test.py::test_cmd_lgif 
test_keeper_four_word_command/test.py::test_cmd_mntr 
[gw0] [  9%] PASSED test_keeper_four_word_command/test.py::test_cmd_mntr 
test_keeper_four_word_command/test.py::test_cmd_rqld 
[gw2] [ 10%] PASSED test_library_bridge/test.py::test_load_all 
test_library_bridge/test.py::test_load_all_many_rows 
[gw3] [ 11%] PASSED test_multi_access_storage_role_management/test.py::test_quotas 
test_multi_access_storage_role_management/test.py::test_role_from_different_storages 
[gw8] [ 12%] PASSED test_log_family_s3/test.py::test_log_family_s3[Log] 
test_log_family_s3/test.py::test_log_family_s3[StripeLog] 
[gw0] [ 13%] PASSED test_keeper_four_word_command/test.py::test_cmd_rqld 
test_keeper_four_word_command/test.py::test_cmd_ruok 
[gw0] [ 14%] PASSED test_keeper_four_word_command/test.py::test_cmd_ruok 
test_keeper_four_word_command/test.py::test_cmd_srst 
[gw0] [ 15%] PASSED test_keeper_four_word_command/test.py::test_cmd_srst 
test_keeper_four_word_command/test.py::test_cmd_srvr 
[gw0] [ 16%] PASSED test_keeper_four_word_command/test.py::test_cmd_srvr 
test_keeper_four_word_command/test.py::test_cmd_stat 
[gw0] [ 17%] PASSED test_keeper_four_word_command/test.py::test_cmd_stat 
test_keeper_four_word_command/test.py::test_cmd_wchc 
[gw0] [ 18%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchc 
test_keeper_four_word_command/test.py::test_cmd_wchp 
[gw5] [ 19%] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 
test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 
[gw0] [ 20%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchp 
test_keeper_four_word_command/test.py::test_cmd_wchs 
[gw0] [ 21%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchs 
[gw8] [ 22%] PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] 
test_log_family_s3/test.py::test_log_family_s3[TinyLog] 
[gw7] [ 23%] PASSED test_keeper_persistent_log/test.py::test_ephemeral_after_restart 
test_keeper_persistent_log/test.py::test_state_after_restart 
[gw8] [ 24%] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] 
test_keeper_session/test.py::test_session_close_shutdown 
[gw5] [ 25%] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 
test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 
[gw5] [ 26%] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 
test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 
[gw4] [ 27%] PASSED test_multiple_disks/test.py::test_alter_double_move_partition[DISK] 
test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME] 
[gw7] [ 28%] PASSED test_keeper_persistent_log/test.py::test_state_after_restart 
test_keeper_persistent_log/test.py::test_state_duplicate_restart 
[gw3] [ 29%] PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages 
test_multi_access_storage_role_management/test.py::test_roles 
[gw6] [ 30%] PASSED test_lost_part/test.py::test_lost_last_part 
test_lost_part/test.py::test_lost_part_mutation 
[gw2] [ 31%] PASSED test_library_bridge/test.py::test_load_all_many_rows 
test_library_bridge/test.py::test_load_ids 
[gw5] [ 32%] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 
[gw2] [ 33%] PASSED test_library_bridge/test.py::test_load_ids 
test_library_bridge/test.py::test_load_keys 
[gw4] [ 34%] PASSED test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME] 
test_multiple_disks/test.py::test_alter_move[mt] 
[gw2] [ 35%] PASSED test_library_bridge/test.py::test_load_keys 
test_library_bridge/test.py::test_null_values 
[gw3] [ 36%] PASSED test_multi_access_storage_role_management/test.py::test_roles 
test_multi_access_storage_role_management/test.py::test_settings_profiles 
[gw2] [ 37%] PASSED test_library_bridge/test.py::test_null_values 
test_library_bridge/test.py::test_path_validation 
[gw1] [ 38%] PASSED test_merge_tree_s3/test.py::test_alter_table_columns[node] 
test_merge_tree_s3/test.py::test_attach_detach_partition[node] 
[gw2] [ 39%] PASSED test_library_bridge/test.py::test_path_validation 
test_library_bridge/test.py::test_recover_after_bridge_crash 
[gw2] [ 40%] PASSED test_library_bridge/test.py::test_recover_after_bridge_crash 
test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive 
[gw3] [ 41%] PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles 
test_multi_access_storage_role_management/test.py::test_users 
[gw1] [ 42%] PASSED test_merge_tree_s3/test.py::test_attach_detach_partition[node] 
test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] 
[gw7] [ 43%] PASSED test_keeper_persistent_log/test.py::test_state_duplicate_restart 
[gw9] [ 44%] PASSED test_max_http_connections_for_replication/test.py::test_keepalive_timeout 
test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count 
test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin 
[gw4] [ 45%] PASSED test_multiple_disks/test.py::test_alter_move[mt] 
test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK] 
[gw3] [ 46%] PASSED test_multi_access_storage_role_management/test.py::test_users 
[gw9] [ 47%] PASSED test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count 
test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count 
test_keeper_internal_secure/test.py::test_secure_raft_works 
test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 
[gw4] [ 48%] PASSED test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK] 
test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME] 
[gw9] [ 49%] PASSED test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count 
[gw0] [ 50%] PASSED test_keeper_session/test.py::test_session_close_shutdown 
test_keeper_session/test.py::test_session_timeout 
[gw0] [ 51%] PASSED test_keeper_session/test.py::test_session_timeout 
test_keeper_force_recovery/test.py::test_cluster_recovery 
[gw2] [ 52%] PASSED test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive 
[gw4] [ 53%] PASSED test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME] 
test_multiple_disks/test.py::test_alter_policy[mt] 
test_keeper_and_access_storage/test.py::test_create_replicated 
test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart 
[gw4] [ 54%] PASSED test_multiple_disks/test.py::test_alter_policy[mt] 
test_multiple_disks/test.py::test_alter_policy[replicated] 
test_keeper_reconfig_add/test.py::test_reconfig_add 
[gw4] [ 55%] PASSED test_multiple_disks/test.py::test_alter_policy[replicated] 
test_multiple_disks/test.py::test_background_move[mt] 
[gw7] [ 56%] PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin 
[gw6] [ 57%] PASSED test_lost_part/test.py::test_lost_part_mutation 
test_lost_part/test.py::test_lost_part_other_replica 
[gw3] [ 58%] PASSED test_keeper_internal_secure/test.py::test_secure_raft_works 
[gw4] [ 59%] PASSED test_multiple_disks/test.py::test_background_move[mt] 
test_keeper_four_word_command/test_allow_list.py::test_allow_list 
test_keeper_nodes_move/test.py::test_node_move 
[gw0] [ 60%] PASSED test_keeper_and_access_storage/test.py::test_create_replicated 
[gw1] [ 61%] PASSED test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] 
test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] 
test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 
[gw1] [ 62%] PASSED test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] 
test_merge_tree_s3/test.py::test_freeze_unfreeze[node] 
[gw2] [ 63%] PASSED test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart 
test_keeper_mntr_pressure/test.py::test_aggressive_mntr 
[gw1] [ 64%] PASSED test_merge_tree_s3/test.py::test_freeze_unfreeze[node] 
test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node] 
[gw0] [ 65%] SKIPPED test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 
[gw7] [ 66%] PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list 
test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 
[gw9] [ 67%] PASSED test_keeper_reconfig_add/test.py::test_reconfig_add 
[gw1] [ 68%] FAILED test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node] 
test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] 
[gw8] [ 69%] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 
test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 
test_keeper_s3_snapshot/test.py::test_s3_upload 
[gw1] [ 70%] PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] 
test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] 
[gw8] [ 71%] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 
test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 
[gw8] [ 72%] SKIPPED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 
[gw1] [ 73%] PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] 
test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] 
[gw7] [ 74%] PASSED test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 
[gw1] [ 75%] PASSED test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] 
test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] 
[gw9] [ 76%] PASSED test_keeper_s3_snapshot/test.py::test_s3_upload 
[gw3] [ 77%] PASSED test_keeper_nodes_move/test.py::test_node_move 
test_keeper_secure_client/test.py::test_connection 
[gw1] [ 78%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] 
test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] 
[gw1] [ 79%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] 
test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] 
[gw5] [ 80%] PASSED test_keeper_force_recovery/test.py::test_cluster_recovery 
[gw1] [ 81%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] 
test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] 
[gw2] [ 82%] PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr 
[gw3] [ 83%] PASSED test_keeper_secure_client/test.py::test_connection 
[gw1] [ 84%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] 
test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] 
test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 
[gw1] [ 85%] PASSED test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] 
test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] 
[gw1] [ 86%] PASSED test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] 
test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] 
[gw1] [ 87%] PASSED test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] 
test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] 
[gw5] [ 88%] PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 
test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view 
[gw5] [ 89%] PASSED test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view 
[gw6] [ 90%] PASSED test_lost_part/test.py::test_lost_part_other_replica 
test_lost_part/test.py::test_lost_part_same_replica 
[gw6] [ 91%] PASSED test_lost_part/test.py::test_lost_part_same_replica 
[gw1] [ 92%] PASSED test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] 
test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] 
[gw1] [ 93%] PASSED test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] 
test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node] 
[gw1] [ 94%] SKIPPED test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node] 
test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node] 
[gw1] [ 95%] SKIPPED test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node] 
test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node] 
[gw1] [ 96%] SKIPPED test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node] 
test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] 
[gw1] [ 97%] PASSED test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] 
test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] 
[gw1] [ 98%] PASSED test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] 
test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] 
[gw1] [ 99%] PASSED test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] 
test_merge_tree_s3/test.py::test_table_manipulations[node] 
[gw1] [100%] PASSED test_merge_tree_s3/test.py::test_table_manipulations[node] 

=================================== FAILURES ===================================
_________________ test_heavy_insert_select_check_memory[node] __________________
[gw1] linux -- Python 3.10.12 /usr/bin/python3

cluster = <helpers.cluster.ClickHouseCluster object at 0x7f847528f5e0>
broken_s3 = <helpers.s3_mocks.broken_s3.MockControl object at 0x7f847523a6b0>
node_name = 'node'

    @pytest.mark.parametrize("node_name", ["node"])
    def test_heavy_insert_select_check_memory(cluster, broken_s3, node_name):
        node = cluster.instances[node_name]
    
        node.query(
            """
            CREATE TABLE central_query_log
            (
                control_plane_id UUID,
                pod_id LowCardinality(String),
                scrape_ts_microseconds DateTime64(6) CODEC(Delta(8), LZ4),
                event_date Date,
                event_time DateTime,
                payload Array(String),
                payload_01 String,
                payload_02 String,
                payload_03 String,
                payload_04 String,
                payload_05 String,
                payload_06 String,
                payload_07 String,
                payload_08 String,
                payload_09 String,
                payload_10 String,
                payload_11 String,
                payload_12 String,
                payload_13 String,
                payload_14 String,
                payload_15 String,
                payload_16 String,
                payload_17 String,
                payload_18 String,
                payload_19 String
            )
            ENGINE=MergeTree()
            PARTITION BY toYYYYMM(event_date)
            ORDER BY (control_plane_id, event_date, pod_id)
            SETTINGS
                storage_policy='s3'
            """
        )
    
        node.query("SYSTEM STOP MERGES central_query_log")
    
        write_count = 2
        write_query_ids = []
        for x in range(write_count):
            query_id = f"INSERT_INTO_TABLE_RANDOM_DATA_QUERY_ID_{x}"
            write_query_ids.append(query_id)
            node.query(
                """
                INSERT INTO central_query_log
                SELECT
                    control_plane_id,
                    pod_id,
                    toStartOfHour(event_time) + toIntervalSecond(randUniform(0,60)) as scrape_ts_microseconds,
                    toDate(event_time) as event_date,
                    event_time,
                    payload,
                    payload[1] as payload_01,
                    payload[2] as payload_02,
                    payload[3] as payload_03,
                    payload[4] as payload_04,
                    payload[5] as payload_05,
                    payload[6] as payload_06,
                    payload[7] as payload_07,
                    payload[8] as payload_08,
                    payload[9] as payload_09,
                    payload[10] as payload_10,
                    payload[11] as payload_11,
                    payload[12] as payload_12,
                    payload[13] as payload_13,
                    payload[14] as payload_14,
                    payload[15] as payload_15,
                    payload[16] as payload_16,
                    payload[17] as payload_17,
                    payload[18] as payload_18,
                    payload[19] as payload_19
                FROM
                (
                    SELECT
                        control_plane_id,
                        substring(payload[1], 1, 5) as pod_id,
                        toDateTime('2022-12-12 00:00:00')
                            + toIntervalDay(floor(randUniform(0,3)))
                            + toIntervalHour(floor(randUniform(0,24)))
                            + toIntervalSecond(floor(randUniform(0,60)))
                            as event_time,
                        payload
                    FROM
                        generateRandom(
                            'control_plane_id UUID, payload Array(String)',
                            NULL,
                            100,
                            100
                        )
                    LIMIT 10000
                )
                SETTINGS
                    max_insert_block_size=256000000,
                    min_insert_block_size_rows=1000000,
                    min_insert_block_size_bytes=256000000
                """,
                query_id=query_id,
            )
    
        memory = 845346116
        for memory_usage, query_id in zip(
            get_memory_usages(node, write_query_ids), write_query_ids
        ):
            assert int(memory_usage) < 1.2 * memory, f"{memory_usage} : {query_id}"
            assert int(memory_usage) > 0.8 * memory, f"{memory_usage} : {query_id}"
    
        broken_s3.setup_slow_answers(minimal_length=1000, timeout=5, count=20)
        broken_s3.setup_fake_multpartuploads()
    
        insert_query_id = f"INSERT_INTO_S3_FUNCTION_QUERY_ID"
        node.query(
            """
            INSERT INTO
                TABLE FUNCTION s3(
                    'http://resolver:8083/root/data/test-upload_{_partition_id}.csv.gz',
                    'minio', 'minio123',
                    'CSV', auto, 'gzip'
                )
            PARTITION BY formatDateTime(subtractHours(toDateTime('2022-12-13 00:00:00'), 1),'%Y-%m-%d_%H:00')
            WITH toDateTime('2022-12-13 00:00:00') as time_point
            SELECT
                *
            FROM central_query_log
            WHERE
                event_date >= subtractDays(toDate(time_point), 1)
                AND scrape_ts_microseconds >= subtractHours(toStartOfHour(time_point), 12)
                AND scrape_ts_microseconds < toStartOfDay(time_point)
            SETTINGS
                s3_max_inflight_parts_for_one_file=1
            """,
            query_id=insert_query_id,
        )
    
        query_id = f"SELECT_QUERY_ID"
        total = node.query(
            """
            SELECT
                count()
            FROM central_query_log
            """,
            query_id=query_id,
        )
        assert int(total) == 10000 * write_count
    
        query_id = f"SELECT_WHERE_QUERY_ID"
        selected = node.query(
            """
            WITH toDateTime('2022-12-13 00:00:00') as time_point
            SELECT
                count()
            FROM central_query_log
            WHERE
                event_date >= subtractDays(toDate(time_point), 1)
                AND scrape_ts_microseconds >= subtractHours(toStartOfHour(time_point), 12)
                AND scrape_ts_microseconds < toStartOfDay(time_point)
            """,
            query_id=query_id,
        )
        assert int(selected) < 4500, selected
        assert int(selected) > 2500, selected
    
        node.query("SYSTEM FLUSH LOGS")
        profile_events = node.query(
            f"""
            SELECT ProfileEvents
                FROM system.query_log
                WHERE query_id='{insert_query_id}'
                AND type='QueryFinish'
            """
        )
    
        memory_usage = get_memory_usage(node, insert_query_id)
        memory = 123507857
        assert int(memory_usage) < 1.2 * memory, f"{memory_usage} {profile_events}"
>       assert int(memory_usage) > 0.8 * memory, f"{memory_usage} {profile_events}"
E       AssertionError: 98073122 {'Query':1,'InsertQuery':1,'QueriesWithSubqueries':2,'SelectQueriesWithSubqueries':2,'FileOpen':162,'ReadBufferFromFileDescriptorRead':162,'ReadBufferFromFileDescriptorReadBytes':8192,'ReadCompressedBytes':69455888,'CompressedReadBufferBlocks':259,'CompressedReadBufferBytes':69353459,'OpenedFileCacheMisses':162,'OpenedFileCacheMicroseconds':2258,'IOBufferAllocs':494,'IOBufferAllocBytes':88733562,'ArenaAllocChunks':1,'ArenaAllocBytes':4096,'FunctionExecute':83,'TableFunctionExecute':1,'MarkCacheMisses':54,'CreatedReadBufferOrdinary':162,'DiskReadElapsedMicroseconds':1264,'NetworkReceiveElapsedMicroseconds':427,'NetworkSendElapsedMicroseconds':4894,'NetworkSendBytes':33033,'InsertedRows':3385,'InsertedBytes':13571649,'SelectedParts':4,'SelectedRanges':4,'SelectedMarks':16,'SelectedRows':20000,'SelectedBytes':80265636,'WaitMarksLoadMicroseconds':176468,'LoadedMarksCount':270,'LoadedMarksMemoryBytes':6168,'ContextLock':501,'ContextLockWaitMicroseconds':12,'RWLockAcquiredReadLocks':4,'PartsLockHoldMicroseconds':68,'PartsLockWaitMicroseconds':1,'RealTimeMicroseconds':30411031,'UserTimeMicroseconds':970146,'SystemTimeMicroseconds':341191,'SoftPageFaults':151062,'OSCPUWaitMicroseconds':48566,'OSCPUVirtualTimeMicroseconds':1311373,'OSWriteBytes':233472,'OSReadChars':42460,'OSWriteChars':240732,'CreatedHTTPConnections':38,'S3ReadMicroseconds':163142,'S3ReadRequestsCount':108,'S3WriteMicroseconds':5128250,'S3WriteRequestsCount':1,'S3WriteRequestsRedirects':1,'DiskS3ReadMicroseconds':163142,'DiskS3ReadRequestsCount':108,'S3PutObject':1,'S3GetObject':108,'DiskS3GetObject':108,'ReadBufferFromS3Microseconds':379916,'ReadBufferFromS3InitMicroseconds':299440,'ReadBufferFromS3Bytes':69457117,'ReadBufferFromS3PreservedSessions':54,'WriteBufferFromS3Microseconds':5162053,'WriteBufferFromS3Bytes':8189166,'WriteBufferFromS3WaitInflightLimitMicroseconds':5162345,'RemoteFSPrefetches':56,'RemoteFSPrefetchedReads':56,'RemoteFSPrefetchedBytes':24496350,'RemoteFSUnprefetchedReads':154,'RemoteFSUnprefetchedBytes':44960767,'RemoteFSBuffers':108,'ThreadpoolReaderTaskMicroseconds':191187,'ThreadpoolReaderReadBytes':24496350,'ThreadpoolReaderSubmit':1154,'AsynchronousRemoteReadWaitMicroseconds':12010,'SynchronousRemoteReadWaitMicroseconds':198156,'LogTest':785,'LogTrace':61,'LogDebug':13,'LogInfo':1}
E         
E       assert 98073122 > (0.8 * 123507857)
E        +  where 98073122 = int(98073122)

test_merge_tree_s3/test.py:1152: AssertionError
------------------------------ Captured log setup ------------------------------
2024-05-01 11:44:00 [ 427 ] INFO : Starting mock server broken_s3.py (mock_servers.py:18, start_mock_servers)
2024-05-01 11:44:00 [ 427 ] DEBUG : run container_id:roottestmergetrees3_resolver_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IGxvZ2dpbmcKaW1wb3J0IHN5cwppbXBvcnQgdGhyZWFkaW5nCmltcG9ydCByYW5kb20KaW1wb3J0IHRpbWUKaW1wb3J0IHVybGxpYi5wYXJzZQppbXBvcnQgaHR0cC5zZXJ2ZXIKaW1wb3J0IHNvY2tldHNlcnZlcgppbXBvcnQgc3RyaW5nCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAoKCklORl9DT1VOVCA9IDEwMDAwMDAwMAoKCmRlZiBfYW5kX3RoZW4odmFsdWUsIGZ1bmMpOgogICAgYXNzZXJ0IGNhbGxhYmxlKGZ1bmMpCiAgICByZXR1cm4gTm9uZSBpZiB2YWx1ZSBpcyBOb25lIGVsc2UgZnVuYyh2YWx1ZSkKCgpjbGFzcyBNb2NrQ29udHJvbDoKICAgIGRlZiBfX2luaXRfXyhzZWxmLCBjbHVzdGVyLCBjb250YWluZXIsIHBvcnQpOgogICAgICAgIHNlbGYuX2NsdXN0ZXIgPSBjbHVzdGVyCiAgICAgICAgc2VsZi5fY29udGFpbmVyID0gY29udGFpbmVyCiAgICAgICAgc2VsZi5fcG9ydCA9IHBvcnQKCiAgICBkZWYgcmVzZXQoc2VsZik6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9yZXNldCIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2FjdGlvbihzZWxmLCB3aGVuLCBjb3VudD1Ob25lLCBhZnRlcj1Ob25lLCBhY3Rpb249Tm9uZSwgYWN0aW9uX2FyZ3M9Tm9uZSk6CiAgICAgICAgdXJsID0gZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3t3aGVufT9ub3RoaW5nPTEiCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgaWYgYWZ0ZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhZnRlcj17YWZ0ZXJ9IgoKICAgICAgICBpZiBhY3Rpb24gaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb249e2FjdGlvbn0iCgogICAgICAgIGlmIGFjdGlvbl9hcmdzIGlzIG5vdCBOb25lOgogICAgICAgICAgICBmb3IgeCBpbiBhY3Rpb25fYXJnczoKICAgICAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb25fYXJncz17eH0iCgogICAgICAgIHJlc3BvbnNlID0gc2VsZi5fY2x1c3Rlci5leGVjX2luX2NvbnRhaW5lcigKICAgICAgICAgICAgc2VsZi5fY2x1c3Rlci5nZXRfY29udGFpbmVyX2lkKHNlbGYuX2NvbnRhaW5lciksCiAgICAgICAgICAgIFsKICAgICAgICAgICAgICAgICJjdXJsIiwKICAgICAgICAgICAgICAgICItcyIsCiAgICAgICAgICAgICAgICB1cmwsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2F0X29iamVjdF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9vYmplY3RfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2F0X3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfcGFydF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2Zha2VfcHV0cyhzZWxmLCBwYXJ0X2xlbmd0aCk6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9mYWtlX3B1dHM/d2hlbl9sZW5ndGhfYmlnZ2VyPXtwYXJ0X2xlbmd0aH0iLAogICAgICAgICAgICBdLAogICAgICAgICAgICBub3Rocm93PVRydWUsCiAgICAgICAgKQogICAgICAgIGFzc2VydCByZXNwb25zZSA9PSAiT0siLCByZXNwb25zZQoKICAgIGRlZiBzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyhzZWxmKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3NldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzPyIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX3Nsb3dfYW5zd2VycygKICAgICAgICBzZWxmLCBtaW5pbWFsX2xlbmd0aD0wLCB0aW1lb3V0PU5vbmUsIHByb2JhYmlsaXR5PU5vbmUsIGNvdW50PU5vbmUKICAgICk6CiAgICAgICAgdXJsID0gKAogICAgICAgICAgICBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9LyIKICAgICAgICAgICAgZiJtb2NrX3NldHRpbmdzL3Nsb3dfcHV0IgogICAgICAgICAgICBmIj9taW5pbWFsX2xlbmd0aD17bWluaW1hbF9sZW5ndGh9IgogICAgICAgICkKCiAgICAgICAgaWYgdGltZW91dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnRpbWVvdXQ9e3RpbWVvdXR9IgoKICAgICAgICBpZiBwcm9iYWJpbGl0eSBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnByb2JhYmlsaXR5PXtwcm9iYWJpbGl0eX0iCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWyJjdXJsIiwgIi1zIiwgdXJsXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCgpjbGFzcyBfU2VydmVyUnVudGltZToKICAgIGNsYXNzIFNsb3dQdXQ6CiAgICAgICAgZGVmIF9faW5pdF9fKAogICAgICAgICAgICBzZWxmLAogICAgICAgICAgICBsb2NrLAogICAgICAgICAgICBwcm9iYWJpbGl0eV89Tm9uZSwKICAgICAgICAgICAgdGltZW91dF89Tm9uZSwKICAgICAgICAgICAgbWluaW1hbF9sZW5ndGhfPU5vbmUsCiAgICAgICAgICAgIGNvdW50Xz1Ob25lLAogICAgICAgICk6CiAgICAgICAgICAgIHNlbGYubG9jayA9IGxvY2sKICAgICAgICAgICAgc2VsZi5wcm9iYWJpbGl0eSA9IHByb2JhYmlsaXR5XyBpZiBwcm9iYWJpbGl0eV8gaXMgbm90IE5vbmUgZWxzZSAxCiAgICAgICAgICAgIHNlbGYudGltZW91dCA9IHRpbWVvdXRfIGlmIHRpbWVvdXRfIGlzIG5vdCBOb25lIGVsc2UgMC4xCiAgICAgICAgICAgIHNlbGYubWluaW1hbF9sZW5ndGggPSBtaW5pbWFsX2xlbmd0aF8gaWYgbWluaW1hbF9sZW5ndGhfIGlzIG5vdCBOb25lIGVsc2UgMAogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAoKICAgICAgICBkZWYgX19zdHJfXyhzZWxmKToKICAgICAgICAgICAgcmV0dXJuICgKICAgICAgICAgICAgICAgIGYicHJvYmFiaWxpdHk6e3NlbGYucHJvYmFiaWxpdHl9IgogICAgICAgICAgICAgICAgZiIgdGltZW91dDp7c2VsZi50aW1lb3V0fSIKICAgICAgICAgICAgICAgIGYiIG1pbmltYWxfbGVuZ3RoOntzZWxmLm1pbmltYWxfbGVuZ3RofSIKICAgICAgICAgICAgICAgIGYiIGNvdW50OntzZWxmLmNvdW50fSIKICAgICAgICAgICAgKQoKICAgICAgICBkZWYgZ2V0X3RpbWVvdXQoc2VsZiwgY29udGVudF9sZW5ndGgpOgogICAgICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgICAgIGlmIGNvbnRlbnRfbGVuZ3RoID4gc2VsZi5taW5pbWFsX2xlbmd0aDoKICAgICAgICAgICAgICAgICAgICBpZiBzZWxmLmNvdW50ID4gMDoKICAgICAgICAgICAgICAgICAgICAgICAgaWYgKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuc2xvd19wdXQucHJvYmFiaWxpdHkgPT0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgb3IgcmFuZG9tLnJhbmRvbSgpIDw9IF9ydW50aW1lLnNsb3dfcHV0LnByb2JhYmlsaXR5CiAgICAgICAgICAgICAgICAgICAgICAgICk6CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZWxmLmNvdW50IC09IDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5zbG93X3B1dC50aW1lb3V0CiAgICAgICAgICAgIHJldHVybiBOb25lCgogICAgY2xhc3MgRXhwZWN0ZWQ1MDBFcnJvckFjdGlvbjoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIGRhdGEgPSAoCiAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4nCiAgICAgICAgICAgICAgICAiPEVycm9yPiIKICAgICAgICAgICAgICAgICI8Q29kZT5FeHBlY3RlZEVycm9yPC9Db2RlPiIKICAgICAgICAgICAgICAgICI8TWVzc2FnZT5tb2NrIHMzIGluamVjdGVkIGVycm9yPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKGRhdGEpCgogICAgY2xhc3MgUmVkaXJlY3RBY3Rpb246CiAgICAgICAgZGVmIF9faW5pdF9fKHNlbGYsIGhvc3Q9ImxvY2FsaG9zdCIsIHBvcnQ9MSk6CiAgICAgICAgICAgIHNlbGYuZHN0X2hvc3QgPSBfYW5kX3RoZW4oaG9zdCwgc3RyKQogICAgICAgICAgICBzZWxmLmRzdF9wb3J0ID0gX2FuZF90aGVuKHBvcnQsIGludCkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVkaXJlY3QoaG9zdD1zZWxmLmRzdF9ob3N0LCBwb3J0PXNlbGYuZHN0X3BvcnQpCgogICAgY2xhc3MgQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uOgogICAgICAgIGRlZiBfX2luaXRfXyhzZWxmLCB3aXRoX3BhcnRpYWxfZGF0YT1Ob25lKToKICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAiIgogICAgICAgICAgICBpZiB3aXRoX3BhcnRpYWxfZGF0YSBpcyBub3QgTm9uZSBhbmQgd2l0aF9wYXJ0aWFsX2RhdGEgPT0gIjEiOgogICAgICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAoCiAgICAgICAgICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICAgICAgICAgIjxJbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgICAgICkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICAgICAgaWYgc2VsZi5wYXJ0aWFsX2RhdGE6CiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMTAwMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuZW5kX2hlYWRlcnMoKQogICAgICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndmaWxlLndyaXRlKGJ5dGVzKHNlbGYucGFydGlhbF9kYXRhLCAiVVRGLTgiKSkKCiAgICAgICAgICAgIHRpbWUuc2xlZXAoMSkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uc2V0c29ja29wdCgKICAgICAgICAgICAgICAgIHNvY2tldC5TT0xfU09DS0VULCBzb2NrZXQuU09fTElOR0VSLCBzdHJ1Y3QucGFjaygiaWkiLCAxLCAwKQogICAgICAgICAgICApCiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5jb25uZWN0aW9uLmNsb3NlKCkKCiAgICBjbGFzcyBCcm9rZW5QaXBlQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgIyBwYXJ0aWFsIHJlYWQKICAgICAgICAgICAgc2VsZi5yZmlsZS5yZWFkKDUwKQoKICAgICAgICAgICAgdGltZS5zbGVlcCgxKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5zZXRzb2Nrb3B0KAogICAgICAgICAgICAgICAgc29ja2V0LlNPTF9TT0NLRVQsIHNvY2tldC5TT19MSU5HRVIsIHN0cnVjdC5wYWNrKCJpaSIsIDEsIDApCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uY2xvc2UoKQoKICAgIGNsYXNzIENvbm5lY3Rpb25SZWZ1c2VkQWN0aW9uKFJlZGlyZWN0QWN0aW9uKToKICAgICAgICBwYXNzCgogICAgY2xhc3MgQ291bnRBZnRlcjoKICAgICAgICBkZWYgX19pbml0X18oCiAgICAgICAgICAgIHNlbGYsIGxvY2ssIGNvdW50Xz1Ob25lLCBhZnRlcl89Tm9uZSwgYWN0aW9uXz1Ob25lLCBhY3Rpb25fYXJnc189W10KICAgICAgICApOgogICAgICAgICAgICBzZWxmLmxvY2sgPSBsb2NrCgogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAogICAgICAgICAgICBzZWxmLmFmdGVyID0gYWZ0ZXJfIGlmIGFmdGVyXyBpcyBub3QgTm9uZSBlbHNlIDAKICAgICAgICAgICAgc2VsZi5hY3Rpb24gPSBhY3Rpb25fCiAgICAgICAgICAgIHNlbGYuYWN0aW9uX2FyZ3MgPSBhY3Rpb25fYXJnc18KCiAgICAgICAgICAgIGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3JlZnVzZWQiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlZnVzZWRBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3Jlc2V0X2J5X3BlZXIiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uKAogICAgICAgICAgICAgICAgICAgICpzZWxmLmFjdGlvbl9hcmdzCiAgICAgICAgICAgICAgICApCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gImJyb2tlbl9waXBlIjoKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkJyb2tlblBpcGVBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJyZWRpcmVjdF90byI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5SZWRpcmVjdEFjdGlvbigqc2VsZi5hY3Rpb25fYXJncykKICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkV4cGVjdGVkNTAwRXJyb3JBY3Rpb24oKQoKICAgICAgICBAc3RhdGljbWV0aG9kCiAgICAgICAgZGVmIGZyb21fY2dpX3BhcmFtcyhsb2NrLCBwYXJhbXMpOgogICAgICAgICAgICByZXR1cm4gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlcigKICAgICAgICAgICAgICAgIGxvY2s9bG9jaywKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICAgICAgYWZ0ZXJfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJhZnRlciIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBhY3Rpb25fPXBhcmFtcy5nZXQoImFjdGlvbiIsIFtOb25lXSlbMF0sCiAgICAgICAgICAgICAgICBhY3Rpb25fYXJnc189cGFyYW1zLmdldCgiYWN0aW9uX2FyZ3MiLCBbXSksCiAgICAgICAgICAgICkKCiAgICAgICAgZGVmIF9fc3RyX18oc2VsZik6CiAgICAgICAgICAgIHJldHVybiBmImNvdW50OntzZWxmLmNvdW50fSBhZnRlcjp7c2VsZi5hZnRlcn0gYWN0aW9uOntzZWxmLmFjdGlvbn0gYWN0aW9uX2FyZ3M6e3NlbGYuYWN0aW9uX2FyZ3N9IgoKICAgICAgICBkZWYgaGFzX2VmZmVjdChzZWxmKToKICAgICAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgICAgICBpZiBzZWxmLmFmdGVyOgogICAgICAgICAgICAgICAgICAgIHNlbGYuYWZ0ZXIgLT0gMQogICAgICAgICAgICAgICAgaWYgc2VsZi5hZnRlciA9PSAwOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQ6CiAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgICAgICAgICAgICAgcmV0dXJuIEZhbHNlCgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyLmluamVjdF9lcnJvcihyZXF1ZXN0X2hhbmRsZXIpCgogICAgZGVmIF9faW5pdF9fKHNlbGYpOgogICAgICAgIHNlbGYubG9jayA9IHRocmVhZGluZy5Mb2NrKCkKICAgICAgICBzZWxmLmF0X3BhcnRfdXBsb2FkID0gTm9uZQogICAgICAgIHNlbGYuYXRfb2JqZWN0X3VwbG9hZCA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfdXBsb2FkcyA9IGRpY3QoKQogICAgICAgIHNlbGYuc2xvd19wdXQgPSBOb25lCiAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgc2VsZi5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgPSBOb25lCgogICAgZGVmIHJlZ2lzdGVyX2Zha2VfdXBsb2FkKHNlbGYsIHVwbG9hZF9pZCwga2V5KToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHNbdXBsb2FkX2lkXSA9IGtleQoKICAgIGRlZiBpc19mYWtlX3VwbG9hZChzZWxmLCB1cGxvYWRfaWQsIGtleSk6CiAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgIGlmIHVwbG9hZF9pZCBpbiBzZWxmLmZha2VfdXBsb2FkczoKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLmZha2VfdXBsb2Fkc1t1cGxvYWRfaWRdID09IGtleQogICAgICAgIHJldHVybiBGYWxzZQoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5hdF9wYXJ0X3VwbG9hZCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5hdF9vYmplY3RfdXBsb2FkID0gTm9uZQogICAgICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHMgPSBkaWN0KCkKICAgICAgICAgICAgc2VsZi5zbG93X3B1dCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgICAgIHNlbGYuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gTm9uZQoKCl9ydW50aW1lID0gX1NlcnZlclJ1bnRpbWUoKQoKCmRlZiBnZXRfcmFuZG9tX3N0cmluZyhsZW5ndGgpOgogICAgIyBjaG9vc2UgZnJvbSBhbGwgbG93ZXJjYXNlIGxldHRlcgogICAgbGV0dGVycyA9IHN0cmluZy5hc2NpaV9sb3dlcmNhc2UKICAgIHJlc3VsdF9zdHIgPSAiIi5qb2luKHJhbmRvbS5jaG9pY2UobGV0dGVycykgZm9yIGkgaW4gcmFuZ2UobGVuZ3RoKSkKICAgIHJldHVybiByZXN1bHRfc3RyCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgX29rKHNlbGYpOgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCgogICAgZGVmIF9waW5nKHNlbGYpOgogICAgICAgIHNlbGYuX29rKCkKCiAgICBkZWYgcmVhZF9hbGxfaW5wdXQoc2VsZik6CiAgICAgICAgY29udGVudF9sZW5ndGggPSBpbnQoc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiLCAwKSkKICAgICAgICB0b19yZWFkID0gY29udGVudF9sZW5ndGgKICAgICAgICB3aGlsZSB0b19yZWFkID4gMDoKICAgICAgICAgICAgIyByZWFkIGNvbnRlbnQgaW4gb3JkZXIgdG8gYXZvaWQgZXJyb3Igb24gY2xpZW50CiAgICAgICAgICAgICMgUG9jbzo6RXhjZXB0aW9uLiBDb2RlOiAxMDAwLCBlLmNvZGUoKSA9IDMyLCBJL08gZXJyb3I6IEJyb2tlbiBwaXBlCiAgICAgICAgICAgICMgZG8gaXQgcGllY2UgYnkgcGllY2UgaW4gb3JkZXIgdG8gYXZvaWQgYmlnIGFsbG9jYXRpb24KICAgICAgICAgICAgc2l6ZSA9IG1pbih0b19yZWFkLCAxMDI0KQogICAgICAgICAgICBzdHIoc2VsZi5yZmlsZS5yZWFkKHNpemUpKQogICAgICAgICAgICB0b19yZWFkIC09IHNpemUKCiAgICBkZWYgcmVkaXJlY3Qoc2VsZiwgaG9zdD1Ob25lLCBwb3J0PU5vbmUpOgogICAgICAgIGlmIGhvc3QgaXMgTm9uZSBhbmQgcG9ydCBpcyBOb25lOgogICAgICAgICAgICBob3N0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1faG9zdAogICAgICAgICAgICBwb3J0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1fcG9ydAoKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDMwNykKICAgICAgICB1cmwgPSBmImh0dHA6Ly97aG9zdH06e3BvcnR9e3NlbGYucGF0aH0iCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgicmVkaXJlY3QgdG8gJXMiLCB1cmwpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiTG9jYXRpb24iLCB1cmwpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIlJlZGlyZWN0ZWQiKQoKICAgIGRlZiB3cml0ZV9lcnJvcihzZWxmLCBkYXRhLCBjb250ZW50X2xlbmd0aD1Ob25lKToKICAgICAgICBpZiBjb250ZW50X2xlbmd0aCBpcyBOb25lOgogICAgICAgICAgICBjb250ZW50X2xlbmd0aCA9IGxlbihkYXRhKQogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoIndyaXRlX2Vycm9yICVzIiwgZGF0YSkKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBzdHIoY29udGVudF9sZW5ndGgpKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIGlmIGRhdGE6CiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3B1dF9vayhzZWxmKToKICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJmYWtlIHB1dCIpCgogICAgICAgIHNlbGYucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiRVRhZyIsICJiNTQzNTdmYWYwNjMyY2NlNDZlOTQyZmE2ODM1NmIzOCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCAwKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBfZmFrZV91cGxvYWRzKHNlbGYsIHBhdGgsIHVwbG9hZF9pZCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8SW5pdGlhdGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPEJ1Y2tldD57YnVja2V0fTwvQnVja2V0PiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+IgogICAgICAgICAgICBmIjxVcGxvYWRJZD57dXBsb2FkX2lkfTwvVXBsb2FkSWQ+IgogICAgICAgICAgICAiPC9Jbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3Bvc3Rfb2soc2VsZiwgcGF0aCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgbG9jYXRpb24gPSAiaHR0cDovL0V4YW1wbGUtQnVja2V0LnMzLlJlZ2lvbi5hbWF6b25hd3MuY29tLyIgKyBwYXRoCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8Q29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPExvY2F0aW9uPntsb2NhdGlvbn08L0xvY2F0aW9uPlxuIgogICAgICAgICAgICBmIjxCdWNrZXQ+e2J1Y2tldH08L0J1Y2tldD5cbiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+XG4iCiAgICAgICAgICAgIGYnPEVUYWc+IjM4NThmNjIyMzBhYzNjOTE1ZjMwMGM2NjQzMTJjMTFmLTkiPC9FVGFnPlxuJwogICAgICAgICAgICBmIjwvQ29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9tb2NrX3NldHRpbmdzKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXRoID0gW3ggZm9yIHggaW4gcGFydHMucGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYXNzZXJ0IHBhdGhbMF0gPT0gIm1vY2tfc2V0dGluZ3MiLCBwYXRoCiAgICAgICAgaWYgbGVuKHBhdGgpIDwgMjoKICAgICAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfcGFydF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCA9IF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKAogICAgICAgICAgICAgICAgX3J1bnRpbWUubG9jaywgcGFyYW1zCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGF0X3BhcnRfdXBsb2FkICVzIiwgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X29iamVjdF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkID0gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoCiAgICAgICAgICAgICAgICBfcnVudGltZS5sb2NrLCBwYXJhbXMKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJzZXQgYXRfb2JqZWN0X3VwbG9hZCAlcyIsIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImZha2VfcHV0cyI6CiAgICAgICAgICAgIHBhcmFtcyA9IHVybGxpYi5wYXJzZS5wYXJzZV9xcyhwYXJ0cy5xdWVyeSwga2VlcF9ibGFua192YWx1ZXM9RmFsc2UpCiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IGludCgKICAgICAgICAgICAgICAgIHBhcmFtcy5nZXQoIndoZW5fbGVuZ3RoX2JpZ2dlciIsIFsxMDI0ICogMTAyNF0pWzBdCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGZha2VfcHV0cyAlcyIsIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAic2xvd19wdXQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5zbG93X3B1dCA9IF9TZXJ2ZXJSdW50aW1lLlNsb3dQdXQoCiAgICAgICAgICAgICAgICBsb2NrPV9ydW50aW1lLmxvY2ssCiAgICAgICAgICAgICAgICBtaW5pbWFsX2xlbmd0aF89X2FuZF90aGVuKHBhcmFtcy5nZXQoIm1pbmltYWxfbGVuZ3RoIiwgW05vbmVdKVswXSwgaW50KSwKICAgICAgICAgICAgICAgIHByb2JhYmlsaXR5Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgicHJvYmFiaWxpdHkiLCBbTm9uZV0pWzBdLCBmbG9hdCksCiAgICAgICAgICAgICAgICB0aW1lb3V0Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgidGltZW91dCIsIFtOb25lXSlbMF0sIGZsb2F0KSwKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBzbG93IHB1dCAlcyIsIF9ydW50aW1lLnNsb3dfcHV0KQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyI6CiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfbXVsdGlwYXJ0X3VwbG9hZCA9IFRydWUKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IHNldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzIikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gKAogICAgICAgICAgICAgICAgX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoX3J1bnRpbWUubG9jaywgcGFyYW1zKQogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAic2V0IGF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCAlcyIsCiAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQsCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAicmVzZXQiOgogICAgICAgICAgICBfcnVudGltZS5yZXNldCgpCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInJlc2V0IikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICBkZWYgZG9fR0VUKHNlbGYpOgogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9waW5nKCkKCiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9tb2NrX3NldHRpbmdzIik6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9tb2NrX3NldHRpbmdzKCkKCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgiZ2V0IHJlZGlyZWN0IikKICAgICAgICByZXR1cm4gc2VsZi5yZWRpcmVjdCgpCgogICAgZGVmIGRvX1BVVChzZWxmKToKICAgICAgICBjb250ZW50X2xlbmd0aCA9IGludChzZWxmLmhlYWRlcnMuZ2V0KCJDb250ZW50LUxlbmd0aCIsIDApKQoKICAgICAgICBpZiBfcnVudGltZS5zbG93X3B1dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdGltZW91dCA9IF9ydW50aW1lLnNsb3dfcHV0LmdldF90aW1lb3V0KGNvbnRlbnRfbGVuZ3RoKQogICAgICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2xvdyBwdXQgJXMiLCB0aW1lb3V0KQogICAgICAgICAgICAgICAgdGltZS5zbGVlcCh0aW1lb3V0KQoKICAgICAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxzcGxpdChzZWxmLnBhdGgpCiAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KCiAgICAgICAgaWYgdXBsb2FkX2lkIGlzIG5vdCBOb25lOgogICAgICAgICAgICBpZiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCBpcyBub3QgTm9uZToKICAgICAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAgICAgInB1dCBhdF9wYXJ0X3VwbG9hZCAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICB1cGxvYWRfaWQsCiAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICApCgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQuaGFzX2VmZmVjdCgpOgogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZC5pbmplY3RfZXJyb3Ioc2VsZikKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuaXNfZmFrZV91cGxvYWQodXBsb2FkX2lkLCBwYXJ0cy5wYXRoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wdXRfb2soKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIGlmIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGVycm9yX2F0X29iamVjdF91cGxvYWQgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQogICAgICAgICAgICBpZiBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBjb250ZW50X2xlbmd0aCA+IF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcjoKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuZmFrZV9wdXRfd2hlbl9sZW5ndGhfYmlnZ2VyLAogICAgICAgICAgICAgICAgICAgICAgICBjb250ZW50X2xlbmd0aCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3B1dF9vaygpCgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICJwdXQgcmVkaXJlY3QgJXMiLAogICAgICAgICAgICBwYXJ0cywKICAgICAgICApCiAgICAgICAgcmV0dXJuIHNlbGYucmVkaXJlY3QoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCiAgICAgICAgdXBsb2FkcyA9IHBhcmFtcy5nZXQoInVwbG9hZHMiLCBbTm9uZV0pWzBdCiAgICAgICAgaWYgdXBsb2FkcyBpcyBub3QgTm9uZToKICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQoKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgdXBsb2FkX2lkID0gZ2V0X3JhbmRvbV9zdHJpbmcoNSkKICAgICAgICAgICAgICAgIF9ydW50aW1lLnJlZ2lzdGVyX2Zha2VfdXBsb2FkKHVwbG9hZF9pZCwgcGFydHMucGF0aCkKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3VwbG9hZHMocGFydHMucGF0aCwgdXBsb2FkX2lkKQoKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KICAgICAgICBpZiBfcnVudGltZS5pc19mYWtlX3VwbG9hZCh1cGxvYWRfaWQsIHBhcnRzLnBhdGgpOgogICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wb3N0X29rKHBhcnRzLnBhdGgpCgogICAgICAgIHJldHVybiBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYucmVkaXJlY3QoKQoKCmNsYXNzIF9UaHJlYWRlZEhUVFBTZXJ2ZXIoc29ja2V0c2VydmVyLlRocmVhZGluZ01peEluLCBodHRwLnNlcnZlci5IVFRQU2VydmVyKToKICAgICIiIkhhbmRsZSByZXF1ZXN0cyBpbiBhIHNlcGFyYXRlIHRocmVhZC4iIiIKCiAgICBkZWYgc2V0X3Vwc3RyZWFtKHNlbGYsIHVwc3RyZWFtX2hvc3QsIHVwc3RyZWFtX3BvcnQpOgogICAgICAgIHNlbGYudXBzdHJlYW1faG9zdCA9IHVwc3RyZWFtX2hvc3QKICAgICAgICBzZWxmLnVwc3RyZWFtX3BvcnQgPSB1cHN0cmVhbV9wb3J0CgoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIjoKICAgIGh0dHBkID0gX1RocmVhZGVkSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCiAgICBpZiBsZW4oc3lzLmFyZ3YpID09IDQ6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKHN5cy5hcmd2WzJdLCBzeXMuYXJndlszXSkKICAgIGVsc2U6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKCJtaW5pbzEiLCA5MDAxKQogICAgaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > broken_s3.py'] (cluster.py:2004, exec_in_container)
2024-05-01 11:44:00 [ 427 ] DEBUG : Command:['docker', 'exec', 'roottestmergetrees3_resolver_1', 'bash', '-c', 'echo aW1wb3J0IGxvZ2dpbmcKaW1wb3J0IHN5cwppbXBvcnQgdGhyZWFkaW5nCmltcG9ydCByYW5kb20KaW1wb3J0IHRpbWUKaW1wb3J0IHVybGxpYi5wYXJzZQppbXBvcnQgaHR0cC5zZXJ2ZXIKaW1wb3J0IHNvY2tldHNlcnZlcgppbXBvcnQgc3RyaW5nCmltcG9ydCBzb2NrZXQKaW1wb3J0IHN0cnVjdAoKCklORl9DT1VOVCA9IDEwMDAwMDAwMAoKCmRlZiBfYW5kX3RoZW4odmFsdWUsIGZ1bmMpOgogICAgYXNzZXJ0IGNhbGxhYmxlKGZ1bmMpCiAgICByZXR1cm4gTm9uZSBpZiB2YWx1ZSBpcyBOb25lIGVsc2UgZnVuYyh2YWx1ZSkKCgpjbGFzcyBNb2NrQ29udHJvbDoKICAgIGRlZiBfX2luaXRfXyhzZWxmLCBjbHVzdGVyLCBjb250YWluZXIsIHBvcnQpOgogICAgICAgIHNlbGYuX2NsdXN0ZXIgPSBjbHVzdGVyCiAgICAgICAgc2VsZi5fY29udGFpbmVyID0gY29udGFpbmVyCiAgICAgICAgc2VsZi5fcG9ydCA9IHBvcnQKCiAgICBkZWYgcmVzZXQoc2VsZik6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9yZXNldCIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2FjdGlvbihzZWxmLCB3aGVuLCBjb3VudD1Ob25lLCBhZnRlcj1Ob25lLCBhY3Rpb249Tm9uZSwgYWN0aW9uX2FyZ3M9Tm9uZSk6CiAgICAgICAgdXJsID0gZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3t3aGVufT9ub3RoaW5nPTEiCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgaWYgYWZ0ZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhZnRlcj17YWZ0ZXJ9IgoKICAgICAgICBpZiBhY3Rpb24gaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb249e2FjdGlvbn0iCgogICAgICAgIGlmIGFjdGlvbl9hcmdzIGlzIG5vdCBOb25lOgogICAgICAgICAgICBmb3IgeCBpbiBhY3Rpb25fYXJnczoKICAgICAgICAgICAgICAgIHVybCArPSBmIiZhY3Rpb25fYXJncz17eH0iCgogICAgICAgIHJlc3BvbnNlID0gc2VsZi5fY2x1c3Rlci5leGVjX2luX2NvbnRhaW5lcigKICAgICAgICAgICAgc2VsZi5fY2x1c3Rlci5nZXRfY29udGFpbmVyX2lkKHNlbGYuX2NvbnRhaW5lciksCiAgICAgICAgICAgIFsKICAgICAgICAgICAgICAgICJjdXJsIiwKICAgICAgICAgICAgICAgICItcyIsCiAgICAgICAgICAgICAgICB1cmwsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2F0X29iamVjdF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9vYmplY3RfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2F0X3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfcGFydF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkKHNlbGYsICoqa3dhcmdzKToKICAgICAgICBzZWxmLnNldHVwX2FjdGlvbigiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIiwgKiprd2FyZ3MpCgogICAgZGVmIHNldHVwX2Zha2VfcHV0cyhzZWxmLCBwYXJ0X2xlbmd0aCk6CiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vbW9ja19zZXR0aW5ncy9mYWtlX3B1dHM/d2hlbl9sZW5ndGhfYmlnZ2VyPXtwYXJ0X2xlbmd0aH0iLAogICAgICAgICAgICBdLAogICAgICAgICAgICBub3Rocm93PVRydWUsCiAgICAgICAgKQogICAgICAgIGFzc2VydCByZXNwb25zZSA9PSAiT0siLCByZXNwb25zZQoKICAgIGRlZiBzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyhzZWxmKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3NldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzPyIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX3Nsb3dfYW5zd2VycygKICAgICAgICBzZWxmLCBtaW5pbWFsX2xlbmd0aD0wLCB0aW1lb3V0PU5vbmUsIHByb2JhYmlsaXR5PU5vbmUsIGNvdW50PU5vbmUKICAgICk6CiAgICAgICAgdXJsID0gKAogICAgICAgICAgICBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9LyIKICAgICAgICAgICAgZiJtb2NrX3NldHRpbmdzL3Nsb3dfcHV0IgogICAgICAgICAgICBmIj9taW5pbWFsX2xlbmd0aD17bWluaW1hbF9sZW5ndGh9IgogICAgICAgICkKCiAgICAgICAgaWYgdGltZW91dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnRpbWVvdXQ9e3RpbWVvdXR9IgoKICAgICAgICBpZiBwcm9iYWJpbGl0eSBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJnByb2JhYmlsaXR5PXtwcm9iYWJpbGl0eX0iCgogICAgICAgIGlmIGNvdW50IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImY291bnQ9e2NvdW50fSIKCiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWyJjdXJsIiwgIi1zIiwgdXJsXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCgpjbGFzcyBfU2VydmVyUnVudGltZToKICAgIGNsYXNzIFNsb3dQdXQ6CiAgICAgICAgZGVmIF9faW5pdF9fKAogICAgICAgICAgICBzZWxmLAogICAgICAgICAgICBsb2NrLAogICAgICAgICAgICBwcm9iYWJpbGl0eV89Tm9uZSwKICAgICAgICAgICAgdGltZW91dF89Tm9uZSwKICAgICAgICAgICAgbWluaW1hbF9sZW5ndGhfPU5vbmUsCiAgICAgICAgICAgIGNvdW50Xz1Ob25lLAogICAgICAgICk6CiAgICAgICAgICAgIHNlbGYubG9jayA9IGxvY2sKICAgICAgICAgICAgc2VsZi5wcm9iYWJpbGl0eSA9IHByb2JhYmlsaXR5XyBpZiBwcm9iYWJpbGl0eV8gaXMgbm90IE5vbmUgZWxzZSAxCiAgICAgICAgICAgIHNlbGYudGltZW91dCA9IHRpbWVvdXRfIGlmIHRpbWVvdXRfIGlzIG5vdCBOb25lIGVsc2UgMC4xCiAgICAgICAgICAgIHNlbGYubWluaW1hbF9sZW5ndGggPSBtaW5pbWFsX2xlbmd0aF8gaWYgbWluaW1hbF9sZW5ndGhfIGlzIG5vdCBOb25lIGVsc2UgMAogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAoKICAgICAgICBkZWYgX19zdHJfXyhzZWxmKToKICAgICAgICAgICAgcmV0dXJuICgKICAgICAgICAgICAgICAgIGYicHJvYmFiaWxpdHk6e3NlbGYucHJvYmFiaWxpdHl9IgogICAgICAgICAgICAgICAgZiIgdGltZW91dDp7c2VsZi50aW1lb3V0fSIKICAgICAgICAgICAgICAgIGYiIG1pbmltYWxfbGVuZ3RoOntzZWxmLm1pbmltYWxfbGVuZ3RofSIKICAgICAgICAgICAgICAgIGYiIGNvdW50OntzZWxmLmNvdW50fSIKICAgICAgICAgICAgKQoKICAgICAgICBkZWYgZ2V0X3RpbWVvdXQoc2VsZiwgY29udGVudF9sZW5ndGgpOgogICAgICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgICAgIGlmIGNvbnRlbnRfbGVuZ3RoID4gc2VsZi5taW5pbWFsX2xlbmd0aDoKICAgICAgICAgICAgICAgICAgICBpZiBzZWxmLmNvdW50ID4gMDoKICAgICAgICAgICAgICAgICAgICAgICAgaWYgKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuc2xvd19wdXQucHJvYmFiaWxpdHkgPT0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgb3IgcmFuZG9tLnJhbmRvbSgpIDw9IF9ydW50aW1lLnNsb3dfcHV0LnByb2JhYmlsaXR5CiAgICAgICAgICAgICAgICAgICAgICAgICk6CiAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZWxmLmNvdW50IC09IDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5zbG93X3B1dC50aW1lb3V0CiAgICAgICAgICAgIHJldHVybiBOb25lCgogICAgY2xhc3MgRXhwZWN0ZWQ1MDBFcnJvckFjdGlvbjoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIGRhdGEgPSAoCiAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4nCiAgICAgICAgICAgICAgICAiPEVycm9yPiIKICAgICAgICAgICAgICAgICI8Q29kZT5FeHBlY3RlZEVycm9yPC9Db2RlPiIKICAgICAgICAgICAgICAgICI8TWVzc2FnZT5tb2NrIHMzIGluamVjdGVkIGVycm9yPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKGRhdGEpCgogICAgY2xhc3MgUmVkaXJlY3RBY3Rpb246CiAgICAgICAgZGVmIF9faW5pdF9fKHNlbGYsIGhvc3Q9ImxvY2FsaG9zdCIsIHBvcnQ9MSk6CiAgICAgICAgICAgIHNlbGYuZHN0X2hvc3QgPSBfYW5kX3RoZW4oaG9zdCwgc3RyKQogICAgICAgICAgICBzZWxmLmRzdF9wb3J0ID0gX2FuZF90aGVuKHBvcnQsIGludCkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVkaXJlY3QoaG9zdD1zZWxmLmRzdF9ob3N0LCBwb3J0PXNlbGYuZHN0X3BvcnQpCgogICAgY2xhc3MgQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uOgogICAgICAgIGRlZiBfX2luaXRfXyhzZWxmLCB3aXRoX3BhcnRpYWxfZGF0YT1Ob25lKToKICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAiIgogICAgICAgICAgICBpZiB3aXRoX3BhcnRpYWxfZGF0YSBpcyBub3QgTm9uZSBhbmQgd2l0aF9wYXJ0aWFsX2RhdGEgPT0gIjEiOgogICAgICAgICAgICAgICAgc2VsZi5wYXJ0aWFsX2RhdGEgPSAoCiAgICAgICAgICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICAgICAgICAgIjxJbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgICAgICkKCiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICAgICAgaWYgc2VsZi5wYXJ0aWFsX2RhdGE6CiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuc2VuZF9oZWFkZXIoIkNvbnRlbnQtTGVuZ3RoIiwgMTAwMDApCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuZW5kX2hlYWRlcnMoKQogICAgICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndmaWxlLndyaXRlKGJ5dGVzKHNlbGYucGFydGlhbF9kYXRhLCAiVVRGLTgiKSkKCiAgICAgICAgICAgIHRpbWUuc2xlZXAoMSkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uc2V0c29ja29wdCgKICAgICAgICAgICAgICAgIHNvY2tldC5TT0xfU09DS0VULCBzb2NrZXQuU09fTElOR0VSLCBzdHJ1Y3QucGFjaygiaWkiLCAxLCAwKQogICAgICAgICAgICApCiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5jb25uZWN0aW9uLmNsb3NlKCkKCiAgICBjbGFzcyBCcm9rZW5QaXBlQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgIyBwYXJ0aWFsIHJlYWQKICAgICAgICAgICAgc2VsZi5yZmlsZS5yZWFkKDUwKQoKICAgICAgICAgICAgdGltZS5zbGVlcCgxKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5zZXRzb2Nrb3B0KAogICAgICAgICAgICAgICAgc29ja2V0LlNPTF9TT0NLRVQsIHNvY2tldC5TT19MSU5HRVIsIHN0cnVjdC5wYWNrKCJpaSIsIDEsIDApCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uY2xvc2UoKQoKICAgIGNsYXNzIENvbm5lY3Rpb25SZWZ1c2VkQWN0aW9uKFJlZGlyZWN0QWN0aW9uKToKICAgICAgICBwYXNzCgogICAgY2xhc3MgQ291bnRBZnRlcjoKICAgICAgICBkZWYgX19pbml0X18oCiAgICAgICAgICAgIHNlbGYsIGxvY2ssIGNvdW50Xz1Ob25lLCBhZnRlcl89Tm9uZSwgYWN0aW9uXz1Ob25lLCBhY3Rpb25fYXJnc189W10KICAgICAgICApOgogICAgICAgICAgICBzZWxmLmxvY2sgPSBsb2NrCgogICAgICAgICAgICBzZWxmLmNvdW50ID0gY291bnRfIGlmIGNvdW50XyBpcyBub3QgTm9uZSBlbHNlIElORl9DT1VOVAogICAgICAgICAgICBzZWxmLmFmdGVyID0gYWZ0ZXJfIGlmIGFmdGVyXyBpcyBub3QgTm9uZSBlbHNlIDAKICAgICAgICAgICAgc2VsZi5hY3Rpb24gPSBhY3Rpb25fCiAgICAgICAgICAgIHNlbGYuYWN0aW9uX2FyZ3MgPSBhY3Rpb25fYXJnc18KCiAgICAgICAgICAgIGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3JlZnVzZWQiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlZnVzZWRBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJjb25uZWN0aW9uX3Jlc2V0X2J5X3BlZXIiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQ29ubmVjdGlvblJlc2V0QnlQZWVyQWN0aW9uKAogICAgICAgICAgICAgICAgICAgICpzZWxmLmFjdGlvbl9hcmdzCiAgICAgICAgICAgICAgICApCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gImJyb2tlbl9waXBlIjoKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkJyb2tlblBpcGVBY3Rpb24oKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJyZWRpcmVjdF90byI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5SZWRpcmVjdEFjdGlvbigqc2VsZi5hY3Rpb25fYXJncykKICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkV4cGVjdGVkNTAwRXJyb3JBY3Rpb24oKQoKICAgICAgICBAc3RhdGljbWV0aG9kCiAgICAgICAgZGVmIGZyb21fY2dpX3BhcmFtcyhsb2NrLCBwYXJhbXMpOgogICAgICAgICAgICByZXR1cm4gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlcigKICAgICAgICAgICAgICAgIGxvY2s9bG9jaywKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICAgICAgYWZ0ZXJfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJhZnRlciIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBhY3Rpb25fPXBhcmFtcy5nZXQoImFjdGlvbiIsIFtOb25lXSlbMF0sCiAgICAgICAgICAgICAgICBhY3Rpb25fYXJnc189cGFyYW1zLmdldCgiYWN0aW9uX2FyZ3MiLCBbXSksCiAgICAgICAgICAgICkKCiAgICAgICAgZGVmIF9fc3RyX18oc2VsZik6CiAgICAgICAgICAgIHJldHVybiBmImNvdW50OntzZWxmLmNvdW50fSBhZnRlcjp7c2VsZi5hZnRlcn0gYWN0aW9uOntzZWxmLmFjdGlvbn0gYWN0aW9uX2FyZ3M6e3NlbGYuYWN0aW9uX2FyZ3N9IgoKICAgICAgICBkZWYgaGFzX2VmZmVjdChzZWxmKToKICAgICAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgICAgICBpZiBzZWxmLmFmdGVyOgogICAgICAgICAgICAgICAgICAgIHNlbGYuYWZ0ZXIgLT0gMQogICAgICAgICAgICAgICAgaWYgc2VsZi5hZnRlciA9PSAwOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQ6CiAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgICAgICAgICAgICAgcmV0dXJuIEZhbHNlCgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyLmluamVjdF9lcnJvcihyZXF1ZXN0X2hhbmRsZXIpCgogICAgZGVmIF9faW5pdF9fKHNlbGYpOgogICAgICAgIHNlbGYubG9jayA9IHRocmVhZGluZy5Mb2NrKCkKICAgICAgICBzZWxmLmF0X3BhcnRfdXBsb2FkID0gTm9uZQogICAgICAgIHNlbGYuYXRfb2JqZWN0X3VwbG9hZCA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfdXBsb2FkcyA9IGRpY3QoKQogICAgICAgIHNlbGYuc2xvd19wdXQgPSBOb25lCiAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgc2VsZi5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgPSBOb25lCgogICAgZGVmIHJlZ2lzdGVyX2Zha2VfdXBsb2FkKHNlbGYsIHVwbG9hZF9pZCwga2V5KToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHNbdXBsb2FkX2lkXSA9IGtleQoKICAgIGRlZiBpc19mYWtlX3VwbG9hZChzZWxmLCB1cGxvYWRfaWQsIGtleSk6CiAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgIGlmIHVwbG9hZF9pZCBpbiBzZWxmLmZha2VfdXBsb2FkczoKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLmZha2VfdXBsb2Fkc1t1cGxvYWRfaWRdID09IGtleQogICAgICAgIHJldHVybiBGYWxzZQoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5hdF9wYXJ0X3VwbG9hZCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5hdF9vYmplY3RfdXBsb2FkID0gTm9uZQogICAgICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHMgPSBkaWN0KCkKICAgICAgICAgICAgc2VsZi5zbG93X3B1dCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgICAgIHNlbGYuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gTm9uZQoKCl9ydW50aW1lID0gX1NlcnZlclJ1bnRpbWUoKQoKCmRlZiBnZXRfcmFuZG9tX3N0cmluZyhsZW5ndGgpOgogICAgIyBjaG9vc2UgZnJvbSBhbGwgbG93ZXJjYXNlIGxldHRlcgogICAgbGV0dGVycyA9IHN0cmluZy5hc2NpaV9sb3dlcmNhc2UKICAgIHJlc3VsdF9zdHIgPSAiIi5qb2luKHJhbmRvbS5jaG9pY2UobGV0dGVycykgZm9yIGkgaW4gcmFuZ2UobGVuZ3RoKSkKICAgIHJldHVybiByZXN1bHRfc3RyCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgX29rKHNlbGYpOgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCgogICAgZGVmIF9waW5nKHNlbGYpOgogICAgICAgIHNlbGYuX29rKCkKCiAgICBkZWYgcmVhZF9hbGxfaW5wdXQoc2VsZik6CiAgICAgICAgY29udGVudF9sZW5ndGggPSBpbnQoc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiLCAwKSkKICAgICAgICB0b19yZWFkID0gY29udGVudF9sZW5ndGgKICAgICAgICB3aGlsZSB0b19yZWFkID4gMDoKICAgICAgICAgICAgIyByZWFkIGNvbnRlbnQgaW4gb3JkZXIgdG8gYXZvaWQgZXJyb3Igb24gY2xpZW50CiAgICAgICAgICAgICMgUG9jbzo6RXhjZXB0aW9uLiBDb2RlOiAxMDAwLCBlLmNvZGUoKSA9IDMyLCBJL08gZXJyb3I6IEJyb2tlbiBwaXBlCiAgICAgICAgICAgICMgZG8gaXQgcGllY2UgYnkgcGllY2UgaW4gb3JkZXIgdG8gYXZvaWQgYmlnIGFsbG9jYXRpb24KICAgICAgICAgICAgc2l6ZSA9IG1pbih0b19yZWFkLCAxMDI0KQogICAgICAgICAgICBzdHIoc2VsZi5yZmlsZS5yZWFkKHNpemUpKQogICAgICAgICAgICB0b19yZWFkIC09IHNpemUKCiAgICBkZWYgcmVkaXJlY3Qoc2VsZiwgaG9zdD1Ob25lLCBwb3J0PU5vbmUpOgogICAgICAgIGlmIGhvc3QgaXMgTm9uZSBhbmQgcG9ydCBpcyBOb25lOgogICAgICAgICAgICBob3N0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1faG9zdAogICAgICAgICAgICBwb3J0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1fcG9ydAoKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDMwNykKICAgICAgICB1cmwgPSBmImh0dHA6Ly97aG9zdH06e3BvcnR9e3NlbGYucGF0aH0iCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgicmVkaXJlY3QgdG8gJXMiLCB1cmwpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiTG9jYXRpb24iLCB1cmwpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIlJlZGlyZWN0ZWQiKQoKICAgIGRlZiB3cml0ZV9lcnJvcihzZWxmLCBkYXRhLCBjb250ZW50X2xlbmd0aD1Ob25lKToKICAgICAgICBpZiBjb250ZW50X2xlbmd0aCBpcyBOb25lOgogICAgICAgICAgICBjb250ZW50X2xlbmd0aCA9IGxlbihkYXRhKQogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoIndyaXRlX2Vycm9yICVzIiwgZGF0YSkKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNTAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBzdHIoY29udGVudF9sZW5ndGgpKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIGlmIGRhdGE6CiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3B1dF9vayhzZWxmKToKICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJmYWtlIHB1dCIpCgogICAgICAgIHNlbGYucmVhZF9hbGxfaW5wdXQoKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiRVRhZyIsICJiNTQzNTdmYWYwNjMyY2NlNDZlOTQyZmE2ODM1NmIzOCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCAwKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQoKICAgIGRlZiBfZmFrZV91cGxvYWRzKHNlbGYsIHBhdGgsIHVwbG9hZF9pZCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8SW5pdGlhdGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPEJ1Y2tldD57YnVja2V0fTwvQnVja2V0PiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+IgogICAgICAgICAgICBmIjxVcGxvYWRJZD57dXBsb2FkX2lkfTwvVXBsb2FkSWQ+IgogICAgICAgICAgICAiPC9Jbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9mYWtlX3Bvc3Rfb2soc2VsZiwgcGF0aCk6CiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHBhcnRzID0gW3ggZm9yIHggaW4gcGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYnVja2V0ID0gcGFydHNbMF0KICAgICAgICBrZXkgPSAiLyIuam9pbihwYXJ0c1sxOl0pCiAgICAgICAgbG9jYXRpb24gPSAiaHR0cDovL0V4YW1wbGUtQnVja2V0LnMzLlJlZ2lvbi5hbWF6b25hd3MuY29tLyIgKyBwYXRoCiAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+XG4nCiAgICAgICAgICAgICI8Q29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgICAgIGYiPExvY2F0aW9uPntsb2NhdGlvbn08L0xvY2F0aW9uPlxuIgogICAgICAgICAgICBmIjxCdWNrZXQ+e2J1Y2tldH08L0J1Y2tldD5cbiIKICAgICAgICAgICAgZiI8S2V5PntrZXl9PC9LZXk+XG4iCiAgICAgICAgICAgIGYnPEVUYWc+IjM4NThmNjIyMzBhYzNjOTE1ZjMwMGM2NjQzMTJjMTFmLTkiPC9FVGFnPlxuJwogICAgICAgICAgICBmIjwvQ29tcGxldGVNdWx0aXBhcnRVcGxvYWRSZXN1bHQ+XG4iCiAgICAgICAgKQoKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoMjAwKQogICAgICAgIHNlbGYuc2VuZF9oZWFkZXIoIkNvbnRlbnQtVHlwZSIsICJ0ZXh0L3htbCIpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4oZGF0YSkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYnl0ZXMoZGF0YSwgIlVURi04IikpCgogICAgZGVmIF9tb2NrX3NldHRpbmdzKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXRoID0gW3ggZm9yIHggaW4gcGFydHMucGF0aC5zcGxpdCgiLyIpIGlmIHhdCiAgICAgICAgYXNzZXJ0IHBhdGhbMF0gPT0gIm1vY2tfc2V0dGluZ3MiLCBwYXRoCiAgICAgICAgaWYgbGVuKHBhdGgpIDwgMjoKICAgICAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfcGFydF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCA9IF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKAogICAgICAgICAgICAgICAgX3J1bnRpbWUubG9jaywgcGFyYW1zCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGF0X3BhcnRfdXBsb2FkICVzIiwgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X29iamVjdF91cGxvYWQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkID0gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoCiAgICAgICAgICAgICAgICBfcnVudGltZS5sb2NrLCBwYXJhbXMKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJzZXQgYXRfb2JqZWN0X3VwbG9hZCAlcyIsIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImZha2VfcHV0cyI6CiAgICAgICAgICAgIHBhcmFtcyA9IHVybGxpYi5wYXJzZS5wYXJzZV9xcyhwYXJ0cy5xdWVyeSwga2VlcF9ibGFua192YWx1ZXM9RmFsc2UpCiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IGludCgKICAgICAgICAgICAgICAgIHBhcmFtcy5nZXQoIndoZW5fbGVuZ3RoX2JpZ2dlciIsIFsxMDI0ICogMTAyNF0pWzBdCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGZha2VfcHV0cyAlcyIsIF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAic2xvd19wdXQiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5zbG93X3B1dCA9IF9TZXJ2ZXJSdW50aW1lLlNsb3dQdXQoCiAgICAgICAgICAgICAgICBsb2NrPV9ydW50aW1lLmxvY2ssCiAgICAgICAgICAgICAgICBtaW5pbWFsX2xlbmd0aF89X2FuZF90aGVuKHBhcmFtcy5nZXQoIm1pbmltYWxfbGVuZ3RoIiwgW05vbmVdKVswXSwgaW50KSwKICAgICAgICAgICAgICAgIHByb2JhYmlsaXR5Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgicHJvYmFiaWxpdHkiLCBbTm9uZV0pWzBdLCBmbG9hdCksCiAgICAgICAgICAgICAgICB0aW1lb3V0Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgidGltZW91dCIsIFtOb25lXSlbMF0sIGZsb2F0KSwKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBzbG93IHB1dCAlcyIsIF9ydW50aW1lLnNsb3dfcHV0KQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyI6CiAgICAgICAgICAgIF9ydW50aW1lLmZha2VfbXVsdGlwYXJ0X3VwbG9hZCA9IFRydWUKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IHNldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzIikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAiYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gKAogICAgICAgICAgICAgICAgX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlci5mcm9tX2NnaV9wYXJhbXMoX3J1bnRpbWUubG9jaywgcGFyYW1zKQogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAic2V0IGF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCAlcyIsCiAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQsCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAicmVzZXQiOgogICAgICAgICAgICBfcnVudGltZS5yZXNldCgpCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInJlc2V0IikKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgcmV0dXJuIHNlbGYud3JpdGVfZXJyb3IoIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICBkZWYgZG9fR0VUKHNlbGYpOgogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9waW5nKCkKCiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9tb2NrX3NldHRpbmdzIik6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9tb2NrX3NldHRpbmdzKCkKCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgiZ2V0IHJlZGlyZWN0IikKICAgICAgICByZXR1cm4gc2VsZi5yZWRpcmVjdCgpCgogICAgZGVmIGRvX1BVVChzZWxmKToKICAgICAgICBjb250ZW50X2xlbmd0aCA9IGludChzZWxmLmhlYWRlcnMuZ2V0KCJDb250ZW50LUxlbmd0aCIsIDApKQoKICAgICAgICBpZiBfcnVudGltZS5zbG93X3B1dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdGltZW91dCA9IF9ydW50aW1lLnNsb3dfcHV0LmdldF90aW1lb3V0KGNvbnRlbnRfbGVuZ3RoKQogICAgICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2xvdyBwdXQgJXMiLCB0aW1lb3V0KQogICAgICAgICAgICAgICAgdGltZS5zbGVlcCh0aW1lb3V0KQoKICAgICAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxzcGxpdChzZWxmLnBhdGgpCiAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KCiAgICAgICAgaWYgdXBsb2FkX2lkIGlzIG5vdCBOb25lOgogICAgICAgICAgICBpZiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCBpcyBub3QgTm9uZToKICAgICAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAgICAgInB1dCBhdF9wYXJ0X3VwbG9hZCAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICB1cGxvYWRfaWQsCiAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICApCgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQuaGFzX2VmZmVjdCgpOgogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZC5pbmplY3RfZXJyb3Ioc2VsZikKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuaXNfZmFrZV91cGxvYWQodXBsb2FkX2lkLCBwYXJ0cy5wYXRoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wdXRfb2soKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIGlmIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGVycm9yX2F0X29iamVjdF91cGxvYWQgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQogICAgICAgICAgICBpZiBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBjb250ZW50X2xlbmd0aCA+IF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcjoKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuZmFrZV9wdXRfd2hlbl9sZW5ndGhfYmlnZ2VyLAogICAgICAgICAgICAgICAgICAgICAgICBjb250ZW50X2xlbmd0aCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3B1dF9vaygpCgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICJwdXQgcmVkaXJlY3QgJXMiLAogICAgICAgICAgICBwYXJ0cywKICAgICAgICApCiAgICAgICAgcmV0dXJuIHNlbGYucmVkaXJlY3QoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCiAgICAgICAgdXBsb2FkcyA9IHBhcmFtcy5nZXQoInVwbG9hZHMiLCBbTm9uZV0pWzBdCiAgICAgICAgaWYgdXBsb2FkcyBpcyBub3QgTm9uZToKICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQoKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgdXBsb2FkX2lkID0gZ2V0X3JhbmRvbV9zdHJpbmcoNSkKICAgICAgICAgICAgICAgIF9ydW50aW1lLnJlZ2lzdGVyX2Zha2VfdXBsb2FkKHVwbG9hZF9pZCwgcGFydHMucGF0aCkKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3VwbG9hZHMocGFydHMucGF0aCwgdXBsb2FkX2lkKQoKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KICAgICAgICBpZiBfcnVudGltZS5pc19mYWtlX3VwbG9hZCh1cGxvYWRfaWQsIHBhcnRzLnBhdGgpOgogICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wb3N0X29rKHBhcnRzLnBhdGgpCgogICAgICAgIHJldHVybiBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYucmVkaXJlY3QoKQoKCmNsYXNzIF9UaHJlYWRlZEhUVFBTZXJ2ZXIoc29ja2V0c2VydmVyLlRocmVhZGluZ01peEluLCBodHRwLnNlcnZlci5IVFRQU2VydmVyKToKICAgICIiIkhhbmRsZSByZXF1ZXN0cyBpbiBhIHNlcGFyYXRlIHRocmVhZC4iIiIKCiAgICBkZWYgc2V0X3Vwc3RyZWFtKHNlbGYsIHVwc3RyZWFtX2hvc3QsIHVwc3RyZWFtX3BvcnQpOgogICAgICAgIHNlbGYudXBzdHJlYW1faG9zdCA9IHVwc3RyZWFtX2hvc3QKICAgICAgICBzZWxmLnVwc3RyZWFtX3BvcnQgPSB1cHN0cmVhbV9wb3J0CgoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIjoKICAgIGh0dHBkID0gX1RocmVhZGVkSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCiAgICBpZiBsZW4oc3lzLmFyZ3YpID09IDQ6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKHN5cy5hcmd2WzJdLCBzeXMuYXJndlszXSkKICAgIGVsc2U6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKCJtaW5pbzEiLCA5MDAxKQogICAgaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > broken_s3.py'] (cluster.py:105, run_and_check)
2024-05-01 11:44:00 [ 427 ] DEBUG : run container_id:roottestmergetrees3_resolver_1 detach:True nothrow:False cmd: ['python', 'broken_s3.py', '8083'] (cluster.py:2004, exec_in_container)
2024-05-01 11:44:00 [ 427 ] DEBUG : run container_id:roottestmergetrees3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2004, exec_in_container)
2024-05-01 11:44:00 [ 427 ] DEBUG : Command:['docker', 'exec', 'roottestmergetrees3_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:105, run_and_check)
2024-05-01 11:44:00 [ 427 ] DEBUG : Exitcode:7 (cluster.py:117, run_and_check)
2024-05-01 11:44:01 [ 427 ] DEBUG : run container_id:roottestmergetrees3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] (cluster.py:2004, exec_in_container)
2024-05-01 11:44:01 [ 427 ] DEBUG : Command:['docker', 'exec', 'roottestmergetrees3_resolver_1', 'curl', '-s', 'http://localhost:8083/'] (cluster.py:105, run_and_check)
2024-05-01 11:44:01 [ 427 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2024-05-01 11:44:01 [ 427 ] DEBUG : broken_s3.py answered OK on attempt 2 (mock_servers.py:53, start_mock_servers)
2024-05-01 11:44:01 [ 427 ] INFO : Mock server broken_s3.py started (mock_servers.py:68, start_mock_servers)
2024-05-01 11:44:01 [ 427 ] DEBUG : http://172.16.9.5:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 (connectionpool.py:546, _make_request)
2024-05-01 11:44:01 [ 427 ] INFO : list_objects (0): [] (test.py:123, list_objects)
2024-05-01 11:44:01 [ 427 ] DEBUG : run container_id:roottestmergetrees3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/mock_settings/reset'] (cluster.py:2004, exec_in_container)
2024-05-01 11:44:01 [ 427 ] DEBUG : Command:['docker', 'exec', 'roottestmergetrees3_resolver_1', 'curl', '-s', 'http://localhost:8083/mock_settings/reset'] (cluster.py:105, run_and_check)
2024-05-01 11:44:01 [ 427 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
------------------------------ Captured log call -------------------------------
2024-05-01 11:44:01 [ 427 ] DEBUG : Executing query 
        CREATE TABLE central_query_log
        (
            control_plane_id UUID,
            pod_id LowCardinality(String),
            scrape_ts_microseconds DateTime64(6) CODEC(Delta(8), LZ4),
            event_date Date,
            event_time DateTime,
            payload Array(String),
            payload_01 String,
            payload_02 String,
            payload_03 String,
            payload_04 String,
            payload_05 String,
            payload_06 String,
            payload_07 String,
            payload_08 String,
            payload_09 String,
            payload_10 String,
            payload_11 String,
            payload_12 String,
            payload_13 String,
            payload_14 String,
            payload_15 String,
            payload_16 String,
            payload_17 String,
            payload_18 String,
            payload_19 String
        )
        ENGINE=MergeTree()
        PARTITION BY toYYYYMM(event_date)
        ORDER BY (control_plane_id, e on node (cluster.py:3423, query)
2024-05-01 11:44:02 [ 427 ] DEBUG : Executing query SYSTEM STOP MERGES central_query_log on node (cluster.py:3423, query)
2024-05-01 11:44:02 [ 427 ] DEBUG : Executing query 
            INSERT INTO central_query_log
            SELECT
                control_plane_id,
                pod_id,
                toStartOfHour(event_time) + toIntervalSecond(randUniform(0,60)) as scrape_ts_microseconds,
                toDate(event_time) as event_date,
                event_time,
                payload,
                payload[1] as payload_01,
                payload[2] as payload_02,
                payload[3] as payload_03,
                payload[4] as payload_04,
                payload[5] as payload_05,
                payload[6] as payload_06,
                payload[7] as payload_07,
                payload[8] as payload_08,
                payload[9] as payload_09,
                payload[10] as payload_10,
                payload[11] as payload_11,
                payload[12] as payload_12,
                payload[13] as payload_13,
                payload[14] as payload_14,
                payload[15] as payload_15,
                payload[16] as pay on node (cluster.py:3423, query)
2024-05-01 11:44:03 [ 427 ] DEBUG : Executing query 
            INSERT INTO central_query_log
            SELECT
                control_plane_id,
                pod_id,
                toStartOfHour(event_time) + toIntervalSecond(randUniform(0,60)) as scrape_ts_microseconds,
                toDate(event_time) as event_date,
                event_time,
                payload,
                payload[1] as payload_01,
                payload[2] as payload_02,
                payload[3] as payload_03,
                payload[4] as payload_04,
                payload[5] as payload_05,
                payload[6] as payload_06,
                payload[7] as payload_07,
                payload[8] as payload_08,
                payload[9] as payload_09,
                payload[10] as payload_10,
                payload[11] as payload_11,
                payload[12] as payload_12,
                payload[13] as payload_13,
                payload[14] as payload_14,
                payload[15] as payload_15,
                payload[16] as pay on node (cluster.py:3423, query)
2024-05-01 11:44:04 [ 427 ] DEBUG : Executing query SYSTEM FLUSH LOGS on node (cluster.py:3423, query)
2024-05-01 11:44:04 [ 427 ] DEBUG : Executing query SELECT memory_usage FROM system.query_log WHERE query_id='INSERT_INTO_TABLE_RANDOM_DATA_QUERY_ID_0'   AND type='QueryFinish' on node (cluster.py:3423, query)
2024-05-01 11:44:05 [ 427 ] DEBUG : Executing query SELECT memory_usage FROM system.query_log WHERE query_id='INSERT_INTO_TABLE_RANDOM_DATA_QUERY_ID_1'   AND type='QueryFinish' on node (cluster.py:3423, query)
2024-05-01 11:44:05 [ 427 ] DEBUG : run container_id:roottestmergetrees3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/mock_settings/slow_put?minimal_length=1000&timeout=5&count=20'] (cluster.py:2004, exec_in_container)
2024-05-01 11:44:05 [ 427 ] DEBUG : Command:['docker', 'exec', 'roottestmergetrees3_resolver_1', 'curl', '-s', 'http://localhost:8083/mock_settings/slow_put?minimal_length=1000&timeout=5&count=20'] (cluster.py:105, run_and_check)
2024-05-01 11:44:05 [ 427 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2024-05-01 11:44:05 [ 427 ] DEBUG : run container_id:roottestmergetrees3_resolver_1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/mock_settings/setup_fake_multpartuploads?'] (cluster.py:2004, exec_in_container)
2024-05-01 11:44:05 [ 427 ] DEBUG : Command:['docker', 'exec', 'roottestmergetrees3_resolver_1', 'curl', '-s', 'http://localhost:8083/mock_settings/setup_fake_multpartuploads?'] (cluster.py:105, run_and_check)
2024-05-01 11:44:05 [ 427 ] DEBUG : Stdout:OK (cluster.py:113, run_and_check)
2024-05-01 11:44:05 [ 427 ] DEBUG : Executing query 
        INSERT INTO
            TABLE FUNCTION s3(
                'http://resolver:8083/root/data/test-upload_{_partition_id}.csv.gz',
                'minio', 'minio123',
                'CSV', auto, 'gzip'
            )
        PARTITION BY formatDateTime(subtractHours(toDateTime('2022-12-13 00:00:00'), 1),'%Y-%m-%d_%H:00')
        WITH toDateTime('2022-12-13 00:00:00') as time_point
        SELECT
            *
        FROM central_query_log
        WHERE
            event_date >= subtractDays(toDate(time_point), 1)
            AND scrape_ts_microseconds >= subtractHours(toStartOfHour(time_point), 12)
            AND scrape_ts_microseconds < toStartOfDay(time_point)
        SETTINGS
            s3_max_inflight_parts_for_one_file=1
         on node (cluster.py:3423, query)
2024-05-01 11:44:11 [ 427 ] DEBUG : Executing query 
        SELECT
            count()
        FROM central_query_log
         on node (cluster.py:3423, query)
2024-05-01 11:44:12 [ 427 ] DEBUG : Executing query 
        WITH toDateTime('2022-12-13 00:00:00') as time_point
        SELECT
            count()
        FROM central_query_log
        WHERE
            event_date >= subtractDays(toDate(time_point), 1)
            AND scrape_ts_microseconds >= subtractHours(toStartOfHour(time_point), 12)
            AND scrape_ts_microseconds < toStartOfDay(time_point)
         on node (cluster.py:3423, query)
2024-05-01 11:44:12 [ 427 ] DEBUG : Executing query SYSTEM FLUSH LOGS on node (cluster.py:3423, query)
2024-05-01 11:44:12 [ 427 ] DEBUG : Executing query 
        SELECT ProfileEvents
            FROM system.query_log
            WHERE query_id='INSERT_INTO_S3_FUNCTION_QUERY_ID'
            AND type='QueryFinish'
         on node (cluster.py:3423, query)
2024-05-01 11:44:13 [ 427 ] DEBUG : Executing query SYSTEM FLUSH LOGS on node (cluster.py:3423, query)
2024-05-01 11:44:13 [ 427 ] DEBUG : Executing query SELECT memory_usage FROM system.query_log WHERE query_id='INSERT_INTO_S3_FUNCTION_QUERY_ID'   AND type='QueryFinish' on node (cluster.py:3423, query)
============================== slowest durations ===============================
306.63s call     test_lost_part/test.py::test_lost_part_other_replica
232.14s call     test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]
60.39s call     test_keeper_force_recovery/test.py::test_cluster_recovery
50.21s call     test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]
48.08s call     test_keeper_mntr_pressure/test.py::test_aggressive_mntr
43.54s call     test_keeper_nodes_move/test.py::test_node_move
37.10s call     test_keeper_force_recovery_single_node/test.py::test_cluster_recovery
33.31s setup    test_max_http_connections_for_replication/test.py::test_keepalive_timeout
32.20s call     test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]
31.73s call     test_lost_part/test.py::test_lost_part_mutation
30.16s setup    test_merge_tree_s3/test.py::test_alter_table_columns[node]
28.57s setup    test_keeper_force_recovery/test.py::test_cluster_recovery
26.40s setup    test_merge_tree_load_parts/test.py::test_merge_tree_load_parts
24.80s setup    test_lost_part/test.py::test_lost_last_part
24.01s setup    test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view
23.61s teardown test_keeper_s3_snapshot/test.py::test_s3_upload
23.50s setup    test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3
23.32s setup    test_multiple_disks/test.py::test_alter_double_move_partition[DISK]
23.00s teardown test_log_family_s3/test.py::test_log_family_s3[TinyLog]
22.90s teardown test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]
22.82s teardown test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3
22.27s teardown test_merge_tree_s3/test.py::test_table_manipulations[node]
21.07s setup    test_keeper_s3_snapshot/test.py::test_s3_upload
20.64s call     test_merge_tree_load_parts/test.py::test_merge_tree_load_parts
19.51s call     test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]
18.93s setup    test_log_family_s3/test.py::test_log_family_s3[Log]
18.83s setup    test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]
17.51s setup    test_keeper_reconfig_add/test.py::test_reconfig_add
17.33s setup    test_keeper_mntr_pressure/test.py::test_aggressive_mntr
17.23s setup    test_keeper_four_word_command/test_allow_list.py::test_allow_list
17.16s setup    test_keeper_four_word_command/test.py::test_cmd_clrs
17.12s setup    test_library_bridge/test.py::test_load_all
16.40s setup    test_keeper_force_recovery_single_node/test.py::test_cluster_recovery
16.30s setup    test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin
16.24s setup    test_keeper_persistent_log/test.py::test_ephemeral_after_restart
16.08s setup    test_keeper_session/test.py::test_session_close_shutdown
15.83s setup    test_multi_access_storage_role_management/test.py::test_quotas
15.76s setup    test_keeper_secure_client/test.py::test_connection
15.44s setup    test_keeper_nodes_move/test.py::test_node_move
15.42s setup    test_keeper_and_access_storage/test.py::test_create_replicated
15.23s call     test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive
14.06s setup    test_keeper_internal_secure/test.py::test_secure_raft_works
13.91s call     test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted
13.20s teardown test_keeper_reconfig_add/test.py::test_reconfig_add
12.82s setup    test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart
12.53s call     test_keeper_persistent_log/test.py::test_state_duplicate_restart
12.19s call     test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node]
11.72s call     test_multiple_disks/test.py::test_alter_move[mt]
10.84s call     test_keeper_session/test.py::test_session_close_shutdown
10.45s call     test_library_bridge/test.py::test_load_all_many_rows
10.40s setup    test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]
9.67s call     test_keeper_reconfig_add/test.py::test_reconfig_add
8.65s call     test_multi_access_storage_role_management/test.py::test_role_from_different_storages
8.41s call     test_max_http_connections_for_replication/test.py::test_keepalive_timeout
8.06s call     test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]
7.87s teardown test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count
7.65s call     test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3
7.18s setup    test_library_bridge/test_exiled.py::test_bridge_dies_with_parent
7.02s call     test_lost_part/test.py::test_lost_part_same_replica
6.69s call     test_keeper_persistent_log/test.py::test_ephemeral_after_restart
6.66s call     test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart
5.87s call     test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]
5.72s teardown test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view
5.41s call     test_multi_access_storage_role_management/test.py::test_roles
5.41s call     test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME]
5.40s teardown test_keeper_internal_secure/test.py::test_secure_raft_works
5.34s teardown test_lost_part/test.py::test_lost_part_same_replica
5.30s call     test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]
5.23s call     test_merge_tree_s3/test.py::test_alter_table_columns[node]
5.16s call     test_merge_tree_s3/test.py::test_attach_detach_partition[node]
5.14s call     test_keeper_persistent_log/test.py::test_state_after_restart
5.03s call     test_multi_access_storage_role_management/test.py::test_settings_profiles
5.02s teardown test_keeper_secure_client/test.py::test_connection
4.95s call     test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK]
4.95s call     test_multi_access_storage_role_management/test.py::test_users
4.92s call     test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]
4.68s call     test_multi_access_storage_role_management/test.py::test_quotas
4.67s call     test_multiple_disks/test.py::test_alter_double_move_partition[DISK]
4.54s call     test_lost_part/test.py::test_lost_last_part
4.53s call     test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME]
4.46s call     test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]
4.28s teardown test_multiple_disks/test.py::test_background_move[mt]
4.16s call     test_keeper_s3_snapshot/test.py::test_s3_upload
4.09s teardown test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin
4.07s call     test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count
3.95s teardown test_keeper_force_recovery/test.py::test_cluster_recovery
3.94s call     test_multiple_disks/test.py::test_background_move[mt]
3.93s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error
3.88s call     test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count
3.80s call     test_merge_tree_s3/test.py::test_freeze_unfreeze[node]
3.71s call     test_merge_tree_s3/test.py::test_table_manipulations[node]
3.48s call     test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]
3.26s teardown test_keeper_session/test.py::test_session_timeout
3.15s teardown test_keeper_mntr_pressure/test.py::test_aggressive_mntr
3.15s call     test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]
3.07s call     test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]
2.93s teardown test_keeper_four_word_command/test.py::test_cmd_wchs
2.92s call     test_library_bridge/test.py::test_load_all
2.87s call     test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]
2.87s call     test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]
2.84s teardown test_keeper_nodes_move/test.py::test_node_move
2.67s call     test_library_bridge/test.py::test_recover_after_bridge_crash
2.58s call     test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]
2.55s teardown test_keeper_four_word_command/test_allow_list.py::test_allow_list
2.55s teardown test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart
2.50s call     test_log_family_s3/test.py::test_log_family_s3[TinyLog]
2.44s teardown test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive
2.42s call     test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]
2.35s call     test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]
2.34s call     test_multiple_disks/test.py::test_alter_policy[mt]
2.33s call     test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]
2.26s teardown test_multi_access_storage_role_management/test.py::test_users
2.19s teardown test_keeper_persistent_log/test.py::test_state_duplicate_restart
2.18s teardown test_keeper_force_recovery_single_node/test.py::test_cluster_recovery
2.12s teardown test_library_bridge/test_exiled.py::test_bridge_dies_with_parent
2.06s teardown test_keeper_and_access_storage/test.py::test_create_replicated
2.05s call     test_log_family_s3/test.py::test_log_family_s3[StripeLog]
1.99s call     test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]
1.94s call     test_multiple_disks/test.py::test_alter_policy[replicated]
1.89s call     test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]
1.80s call     test_log_family_s3/test.py::test_log_family_s3[Log]
1.77s call     test_library_bridge/test.py::test_load_ids
1.68s call     test_keeper_four_word_command/test_allow_list.py::test_allow_list
1.66s call     test_library_bridge/test.py::test_path_validation
1.44s call     test_library_bridge/test.py::test_load_keys
1.42s setup    test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node]
1.03s call     test_keeper_four_word_command/test.py::test_cmd_rqld
0.86s call     test_library_bridge/test.py::test_null_values
0.85s setup    test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count
0.75s call     test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view
0.70s call     test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin
0.68s call     test_keeper_four_word_command/test.py::test_cmd_mntr
0.62s call     test_keeper_secure_client/test.py::test_connection
0.60s call     test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]
0.56s call     test_keeper_four_word_command/test.py::test_cmd_lgif
0.43s call     test_library_bridge/test_exiled.py::test_bridge_dies_with_parent
0.34s call     test_keeper_internal_secure/test.py::test_secure_raft_works
0.34s call     test_keeper_session/test.py::test_session_timeout
0.33s call     test_keeper_four_word_command/test.py::test_cmd_crst
0.31s call     test_keeper_four_word_command/test.py::test_cmd_srvr
0.29s call     test_keeper_four_word_command/test.py::test_cmd_dump
0.29s call     test_keeper_four_word_command/test.py::test_cmd_wchp
0.28s call     test_keeper_four_word_command/test.py::test_cmd_stat
0.26s call     test_keeper_four_word_command/test.py::test_cmd_wchc
0.26s call     test_keeper_four_word_command/test.py::test_cmd_cons
0.25s call     test_keeper_four_word_command/test.py::test_cmd_wchs
0.22s call     test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node]
0.22s call     test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node]
0.22s call     test_keeper_and_access_storage/test.py::test_create_replicated
0.22s call     test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error
0.22s call     test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node]
0.22s call     test_keeper_four_word_command/test.py::test_cmd_clrs
0.16s call     test_keeper_four_word_command/test.py::test_cmd_srst
0.13s call     test_keeper_four_word_command/test.py::test_cmd_conf
0.10s call     test_keeper_four_word_command/test.py::test_cmd_csnp
0.07s setup    test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node]
0.07s setup    test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node]
0.06s setup    test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node]
0.06s setup    test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]
0.06s setup    test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]
0.06s setup    test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]
0.06s setup    test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]
0.02s call     test_keeper_four_word_command/test.py::test_cmd_isro
0.01s call     test_keeper_four_word_command/test.py::test_cmd_ruok
0.01s setup    test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]
0.00s setup    test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]
0.00s setup    test_merge_tree_s3/test.py::test_table_manipulations[node]
0.00s setup    test_merge_tree_s3/test.py::test_attach_detach_partition[node]
0.00s setup    test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]
0.00s setup    test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]
0.00s setup    test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]
0.00s setup    test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]
0.00s setup    test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]
0.00s setup    test_merge_tree_s3/test.py::test_freeze_unfreeze[node]
0.00s setup    test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]
0.00s setup    test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]
0.00s setup    test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]
0.00s setup    test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]
0.00s setup    test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]
0.00s teardown test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]
0.00s teardown test_multiple_disks/test.py::test_alter_double_move_partition[DISK]
0.00s teardown test_keeper_persistent_log/test.py::test_state_after_restart
0.00s teardown test_keeper_persistent_log/test.py::test_ephemeral_after_restart
0.00s setup    test_log_family_s3/test.py::test_log_family_s3[StripeLog]
0.00s teardown test_merge_tree_s3/test.py::test_alter_table_columns[node]
0.00s setup    test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]
0.00s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts
0.00s setup    test_multiple_disks/test.py::test_alter_policy[mt]
0.00s teardown test_log_family_s3/test.py::test_log_family_s3[Log]
0.00s setup    test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]
0.00s teardown test_lost_part/test.py::test_lost_last_part
0.00s setup    test_multi_access_storage_role_management/test.py::test_users
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_lgif
0.00s setup    test_multi_access_storage_role_management/test.py::test_settings_profiles
0.00s setup    test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive
0.00s teardown test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]
0.00s setup    test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]
0.00s setup    test_library_bridge/test.py::test_load_all_many_rows
0.00s setup    test_log_family_s3/test.py::test_log_family_s3[TinyLog]
0.00s setup    test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME]
0.00s setup    test_keeper_session/test.py::test_session_timeout
0.00s teardown test_keeper_session/test.py::test_session_close_shutdown
0.00s teardown test_multiple_disks/test.py::test_alter_move[mt]
0.00s teardown test_library_bridge/test.py::test_load_all
0.00s teardown test_max_http_connections_for_replication/test.py::test_keepalive_timeout
0.00s setup    test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK]
0.00s teardown test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_mntr
0.00s setup    test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_wchs
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchp
0.00s setup    test_keeper_persistent_log/test.py::test_state_duplicate_restart
0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]
0.00s teardown test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]
0.00s teardown test_lost_part/test.py::test_lost_part_other_replica
0.00s setup    test_multiple_disks/test.py::test_alter_move[mt]
0.00s teardown test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME]
0.00s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted
0.00s teardown test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_wchp
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_cons
0.00s setup    test_library_bridge/test.py::test_load_keys
0.00s setup    test_library_bridge/test.py::test_path_validation
0.00s setup    test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchc
0.00s teardown test_library_bridge/test.py::test_load_ids
0.00s setup    test_multiple_disks/test.py::test_alter_policy[replicated]
0.00s teardown test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]
0.00s setup    test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error
0.00s setup    test_multiple_disks/test.py::test_background_move[mt]
0.00s setup    test_library_bridge/test.py::test_recover_after_bridge_crash
0.00s teardown test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node]
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_crst
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_ruok
0.00s teardown test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_mntr
0.00s teardown test_library_bridge/test.py::test_null_values
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_stat
0.00s teardown test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node]
0.00s setup    test_library_bridge/test.py::test_null_values
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_dump
0.00s teardown test_merge_tree_s3/test.py::test_attach_detach_partition[node]
0.00s teardown test_multi_access_storage_role_management/test.py::test_settings_profiles
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_crst
0.00s setup    test_library_bridge/test.py::test_load_ids
0.00s teardown test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node]
0.00s setup    test_lost_part/test.py::test_lost_part_same_replica
0.00s teardown test_library_bridge/test.py::test_path_validation
0.00s teardown test_multi_access_storage_role_management/test.py::test_role_from_different_storages
0.00s setup    test_lost_part/test.py::test_lost_part_mutation
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srvr
0.00s teardown test_library_bridge/test.py::test_load_keys
0.00s teardown test_merge_tree_s3/test.py::test_freeze_unfreeze[node]
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_clrs
0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]
0.00s teardown test_multiple_disks/test.py::test_alter_policy[mt]
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_isro
0.00s setup    test_multi_access_storage_role_management/test.py::test_role_from_different_storages
0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]
0.00s teardown test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]
0.00s teardown test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]
0.00s setup    test_multi_access_storage_role_management/test.py::test_roles
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_lgif
0.00s teardown test_multi_access_storage_role_management/test.py::test_quotas
0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]
0.00s teardown test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_srst
0.00s teardown test_library_bridge/test.py::test_recover_after_bridge_crash
0.00s teardown test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]
0.00s teardown test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]
0.00s teardown test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME]
0.00s teardown test_multiple_disks/test.py::test_alter_policy[replicated]
0.00s teardown test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]
0.00s teardown test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]
0.00s teardown test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_stat
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_isro
0.00s teardown test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_wchc
0.00s teardown test_log_family_s3/test.py::test_log_family_s3[StripeLog]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_cons
0.00s teardown test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_csnp
0.00s setup    test_keeper_persistent_log/test.py::test_state_after_restart
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_rqld
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_conf
0.00s teardown test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srst
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_srvr
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_dump
0.00s teardown test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_rqld
0.00s setup    test_keeper_four_word_command/test.py::test_cmd_conf
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_csnp
0.00s teardown test_multi_access_storage_role_management/test.py::test_roles
0.00s setup    test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count
0.00s teardown test_keeper_four_word_command/test.py::test_cmd_ruok
0.00s setup    test_lost_part/test.py::test_lost_part_other_replica
0.00s teardown test_library_bridge/test.py::test_load_all_many_rows
0.00s teardown test_lost_part/test.py::test_lost_part_mutation
=========================== short test summary info ============================
FAILED test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node]
PASSED test_keeper_four_word_command/test.py::test_cmd_clrs
PASSED test_keeper_four_word_command/test.py::test_cmd_conf
PASSED test_keeper_four_word_command/test.py::test_cmd_cons
PASSED test_keeper_four_word_command/test.py::test_cmd_crst
PASSED test_keeper_four_word_command/test.py::test_cmd_csnp
PASSED test_keeper_four_word_command/test.py::test_cmd_dump
PASSED test_keeper_four_word_command/test.py::test_cmd_isro
PASSED test_keeper_four_word_command/test.py::test_cmd_lgif
PASSED test_keeper_four_word_command/test.py::test_cmd_mntr
PASSED test_library_bridge/test.py::test_load_all
PASSED test_multi_access_storage_role_management/test.py::test_quotas
PASSED test_log_family_s3/test.py::test_log_family_s3[Log]
PASSED test_keeper_four_word_command/test.py::test_cmd_rqld
PASSED test_keeper_four_word_command/test.py::test_cmd_ruok
PASSED test_keeper_four_word_command/test.py::test_cmd_srst
PASSED test_keeper_four_word_command/test.py::test_cmd_srvr
PASSED test_keeper_four_word_command/test.py::test_cmd_stat
PASSED test_keeper_four_word_command/test.py::test_cmd_wchc
PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]
PASSED test_keeper_four_word_command/test.py::test_cmd_wchp
PASSED test_keeper_four_word_command/test.py::test_cmd_wchs
PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog]
PASSED test_keeper_persistent_log/test.py::test_ephemeral_after_restart
PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog]
PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]
PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]
PASSED test_multiple_disks/test.py::test_alter_double_move_partition[DISK]
PASSED test_keeper_persistent_log/test.py::test_state_after_restart
PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages
PASSED test_lost_part/test.py::test_lost_last_part
PASSED test_library_bridge/test.py::test_load_all_many_rows
PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]
PASSED test_library_bridge/test.py::test_load_ids
PASSED test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME]
PASSED test_library_bridge/test.py::test_load_keys
PASSED test_multi_access_storage_role_management/test.py::test_roles
PASSED test_library_bridge/test.py::test_null_values
PASSED test_merge_tree_s3/test.py::test_alter_table_columns[node]
PASSED test_library_bridge/test.py::test_path_validation
PASSED test_library_bridge/test.py::test_recover_after_bridge_crash
PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles
PASSED test_merge_tree_s3/test.py::test_attach_detach_partition[node]
PASSED test_keeper_persistent_log/test.py::test_state_duplicate_restart
PASSED test_max_http_connections_for_replication/test.py::test_keepalive_timeout
PASSED test_multiple_disks/test.py::test_alter_move[mt]
PASSED test_multi_access_storage_role_management/test.py::test_users
PASSED test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count
PASSED test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK]
PASSED test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count
PASSED test_keeper_session/test.py::test_session_close_shutdown
PASSED test_keeper_session/test.py::test_session_timeout
PASSED test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive
PASSED test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME]
PASSED test_multiple_disks/test.py::test_alter_policy[mt]
PASSED test_multiple_disks/test.py::test_alter_policy[replicated]
PASSED test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin
PASSED test_lost_part/test.py::test_lost_part_mutation
PASSED test_keeper_internal_secure/test.py::test_secure_raft_works
PASSED test_multiple_disks/test.py::test_background_move[mt]
PASSED test_keeper_and_access_storage/test.py::test_create_replicated
PASSED test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]
PASSED test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]
PASSED test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart
PASSED test_merge_tree_s3/test.py::test_freeze_unfreeze[node]
PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list
PASSED test_keeper_reconfig_add/test.py::test_reconfig_add
PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts
PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]
PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted
PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]
PASSED test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3
PASSED test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]
PASSED test_keeper_s3_snapshot/test.py::test_s3_upload
PASSED test_keeper_nodes_move/test.py::test_node_move
PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]
PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]
PASSED test_keeper_force_recovery/test.py::test_cluster_recovery
PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]
PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr
PASSED test_keeper_secure_client/test.py::test_connection
PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]
PASSED test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]
PASSED test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]
PASSED test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]
PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery
PASSED test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view
PASSED test_lost_part/test.py::test_lost_part_other_replica
PASSED test_lost_part/test.py::test_lost_part_same_replica
PASSED test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]
PASSED test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]
PASSED test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]
PASSED test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]
PASSED test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]
PASSED test_merge_tree_s3/test.py::test_table_manipulations[node]
SKIPPED [1] test_library_bridge/test_exiled.py:53: Leak sanitizer falsely reports about a leak of 16 bytes in clickhouse-odbc-bridge
SKIPPED [1] test_merge_tree_load_parts/test.py:209: Skip with debug build and sanitizers.             This test intentionally triggers LOGICAL_ERROR which leads to crash with those builds
SKIPPED [3] test_merge_tree_s3/test.py:861: Disabled for sanitizers
============= 1 failed, 94 passed, 5 skipped in 477.19s (0:07:57) ==============
Traceback (most recent call last):
  File "/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration/./runner", line 455, in <module>
    subprocess.check_call(cmd, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker run   --rm --name clickhouse_integration_tests_uuiajx --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_BASE_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_KERBERIZED_HADOOP_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_KERBEROS_KDC_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_JS_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_NGINX_DAV_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-8e9be16153250c8307730c9b88c8517181f494ce   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600  -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_jbod_load_balancing/test.py::test_jbod_load_balancing_round_robin test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_force_recovery/test.py::test_cluster_recovery test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_internal_secure/test.py::test_secure_raft_works test_keeper_mntr_data_size/test.py::test_mntr_data_size_after_restart test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_persistent_log/test.py::test_ephemeral_after_restart test_keeper_persistent_log/test.py::test_state_after_restart test_keeper_persistent_log/test.py::test_state_duplicate_restart test_keeper_reconfig_add/test.py::test_reconfig_add test_keeper_restore_from_snapshot/test_disk_s3.py::test_recover_from_snapshot_with_disk_s3 test_keeper_s3_snapshot/test.py::test_s3_upload test_keeper_secure_client/test.py::test_connection test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_library_bridge/test.py::test_load_all test_library_bridge/test.py::test_load_all_many_rows test_library_bridge/test.py::test_load_ids test_library_bridge/test.py::test_load_keys test_library_bridge/test.py::test_null_values test_library_bridge/test.py::test_path_validation test_library_bridge/test.py::test_recover_after_bridge_crash test_library_bridge/test.py::test_server_restart_bridge_might_be_stil_alive test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_lost_part/test.py::test_lost_last_part test_lost_part/test.py::test_lost_part_mutation test_lost_part/test.py::test_lost_part_other_replica test_lost_part/test.py::test_lost_part_same_replica test_max_http_connections_for_replication/test.py::test_keepalive_timeout test_max_http_connections_for_replication/test.py::test_multiple_endpoint_connections_count test_max_http_connections_for_replication/test.py::test_single_endpoint_connections_count test_max_rows_to_read_leaf_with_view/test.py::test_max_rows_to_read_leaf_via_view test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 'test_merge_tree_s3/test.py::test_alter_table_columns[node]' 'test_merge_tree_s3/test.py::test_attach_detach_partition[node]' 'test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]' 'test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]' 'test_merge_tree_s3/test.py::test_freeze_unfreeze[node]' 'test_merge_tree_s3/test.py::test_heavy_insert_select_check_memory[node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]' 'test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]' 'test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]' 'test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]' 'test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]' 'test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]' 'test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node]' 'test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]' 'test_merge_tree_s3/test.py::test_table_manipulations[node]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]' test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users 'test_multiple_disks/test.py::test_alter_double_move_partition[DISK]' 'test_multiple_disks/test.py::test_alter_double_move_partition[VOLUME]' 'test_multiple_disks/test.py::test_alter_move[mt]' 'test_multiple_disks/test.py::test_alter_move_half_of_partition[DISK]' 'test_multiple_disks/test.py::test_alter_move_half_of_partition[VOLUME]' 'test_multiple_disks/test.py::test_alter_policy[mt]' 'test_multiple_disks/test.py::test_alter_policy[replicated]' 'test_multiple_disks/test.py::test_background_move[mt]'  -vvv" altinityinfra/integration-tests-runner:0-8e9be16153250c8307730c9b88c8517181f494ce ' returned non-zero exit status 1.